Nightcrawler's Message Board - Project - Text Angel (GUI Generic Dumper/Inserter)

Welcome, Guest. Please Login or Register

	The Purple Parade is marching in full stride to the beat of that 'other' drum we all hear, but generally ignore.

Nightcrawler's Message Board › Rom Hacking/Translation Board › Rom Hacking/Translation Board › Project - Text Angel (GUI Generic Dumper/Inserter)

(Moderator: Nightcrawler)

‹ No Topic | Next Topic ›

Pages: 1 2

Project - Text Angel (GUI Generic Dumper/Inserter) (Read 28579 times)

Nightcrawler

YaBB Administrator

Offline

The Dark Angel of Romhacking

Posts: 3251
USA
Gender: male

Project - Text Angel (GUI Generic Dumper/Inserter)
May 12^th, 2010 at 3:03pm

Overview:
In pursuit of working smarter and not harder I had started toying around with the idea of new GUI based generic dumper/inserter. The aim for the dumper was originally be able to do everything that Cartographer could do and then some. I was trying to eliminate the need for a custom dumper in as many cases as possible. On the insertion side, it would ideally be able to insert everything that it can dump. I thought Cartographer and Atlas made some good progress in that direction, but lacked simplicity and ease of use.

Along the way, as features grew, this utility blossomed into a full project oriented utility. It is now capable of handling many different types of text and pointer structures and hopefully handle most needs for your entire project from dialog to item lists. It's the most powerful tool I've ever programmed. The utility uses a project and string oriented approach. Start a new project, add all the files you need. Then add the necessary pointer tables, string blocks, etc. Now you can go to work and translate them right in the utility, or you can dump to full Atlas compliant text output, or even export to XML!

Probably even more important than what the utility currently supports, is the potential of what it can support. With a flexible framework in place and a project oriented interface, the utility can continue to grow and expand to cover many needs. Exciting feature ideas such as integrated script formatting, in-game screen previews, flexible data structure definitions, virtual file systems, and more are possible down the road. Let's not jump too far ahead for. For now, here's what we have so far implemented today!

Images:
http://transcorp.romhacking.net/TextAngel/TextAngel1.png
http://transcorp.romhacking.net/TextAngel/TextAngel2.png
http://transcorp.romhacking.net/TextAngel/TextAngel3.png
http://transcorp.romhacking.net/TextAngel/TextAngel4.png

Features:

Batch File Support - Automated dumping across many files.
Table Standard 1.0 WIP draft support.
Raw Start/Stop Dumps
Pointer Table based Dumps (Also can ignore invalid pointers if desired)
Pointer List based Dumps for scattered pointers.
LOROM00, LOROM80, HIROM, GB, LINEAR modes
8-bit, 16-bit, 24-bit, 32-bit pointer support
Little and Big Endian
Pointer Spacing (Spacing between Pointers)
Pointer per byte spacing (arbitrary Spacing between individual bytes in a pointer)
Relative Pointers with Table Start as Base
Relative Pointers with Pointer Location as Base.
Pointer Global Offset
Header support for pointer calculations.
AutoDiscovery/Pointer Scan based dumps
Atlas pointer compatibility
Full Atlas Ready output out of the box. (Currently having issues as this can complicated for some of the complicated features and structures TextAngel supports.)
EndTerminated Strings
Pascal Strings
Fixed Length Strings
Fixed Length Strings Based on Pointer Calculations (size=pointer(n+1) - Pointer (n))
Fixed Length Lines
Dictionary Dump (Uses string prefix/postfix ability)
Re-ordering dump by string location
Duplicate string detection
Overlapping Strings detection
XML Export
Dump Templates
Advanced Tree view with dump preview
UTF-8, S-JIS, ASCII, and JP-EUC encoding support for tables and script dump.
Project support for dumping all areas of your project. (script, items, menu etc.)

« Last Edit: Jun 3^rd, 2013 at 3:57pm by Nightcrawler »

ROMhacking.net - The central hub of the ROM hacking community.

IP Logged

Nightcrawler

YaBB Administrator

Offline

The Dark Angel of Romhacking

Posts: 3251
USA
Gender: male

Re: GUI Based Generic Script Dumper
Reply #1 - Jun 10^th, 2010 at 2:48pm

I spent too much time on a decent C# MVP (model-view-presenter) pattern based application framework. It's a struggle to be a good software architect when you're on the learn it yourself program with no formal education or mentor on the matter. I'm still not sure how well the framework will scale, but I gave it a best shot. My error manager and handling framework needs some work for sure, but I'm not really sure how to do it better. Any C# professionals out there with MVP design pattern experience?

I started beefing up my table library to support all the features that have been discussed for the standard. Supports flexible formatting (\n\r\t), end tokens, linked entries, and UTF-8 just fine.

I've been been testing it and it's working well so far. It ignores lines it doesn't recognize as supported features and errors out on those that should be supported, but are malformed and unable to be parsed. I thought that was a better approach than spitting out errors on any table that might contain a dump bookmark or other unsupported feature. Both Unix and Windows newlines are supported OK too I think.

I am working on incorporating support for Atlas's address modes (LoROM00, LoROM80, HiROM, GB, Linear).

I'm likely going to have selectable encoding so scripts/tables can be ASCII, S-JIS, UTF-8, or EUC. UTF-8 will be the default.

I've got a host of other potential features I'd like to have for the dumper when I get into it soon. But, I'm going to start small. I'm not sure how much time I want to invest in this.

While it will have Atlas compatible output, I'm trying to have an Inserter built-in as well. Though figuring out the logistics for an equally flexible inserter on par with Atlas is difficult. It may end up with Atlas only output at first and decide if it's worthwhile to develop further.

So, things are moving along. I'd like to get back to progress on Tenshi soon. I haven't been able to touch it since March which spurred my need for better 'universal' approaches and tools.

ROMhacking.net - The central hub of the ROM hacking community.

IP Logged

DaMarsMan Nobleman Offline Posts: 163	Re: GUI Based Generic Script Dumper Reply #2 - Jul 12^th, 2010 at 1:31pm How about an option for pointer scan? Sometimes a script is in order but the pointer table is completely out of order or broken up. Also, can you have an "Add to" box for the pointers? Many systems don't use standard pointers. Also, the ability to save dump settings as XML would rock.
Back to top	Dragon Quest 5 for ps2 hacker/translator....&& IP Logged

Nightcrawler

YaBB Administrator

Offline

The Dark Angel of Romhacking

Posts: 3251
USA
Gender: male

Re: GUI Based Generic Script Dumper
Reply #3 - Jul 12^th, 2010 at 3:26pm

DaMarsMan wrote on Jul 12^th, 2010 at 1:31pm:

How about an option for pointer scan? Sometimes a script is in order but the pointer table is completely out of order or broken up.

I'm not following. What would a pointer scan do? Quick Example?

Quote:

Also, can you have an "Add to" box for the pointers? Many systems don't use standard pointers.

Isn't this the same thing as the Global Offset Box? I could go into more detail with an example. All pointers are added to the global offset. If the pointers are relative, you'd set the global offset to the pointer table start. If the global pointer is 0, the pointers are effectively absolute. Then, there's the option to use the pointer location as the base.

Again, we'd need to discuss an example that you think might not be covered.

Quote:

Also, the ability to save dump settings as XML would rock.

Indeed. Already planned.

I'd also like to have it so you could point to one of those config files and you could run it from the command line that way as well. Certainly, there will be instances where one might want to run from batch file.

ROMhacking.net - The central hub of the ROM hacking community.

IP Logged

DaMarsMan

Nobleman

Offline

Posts: 163

Re: GUI Based Generic Script Dumper
Reply #4 - Jul 12^th, 2010 at 6:14pm

Quote:

I'm not following. What would a pointer scan do? Quick Example?

Not all pointers are within a table. Sometimes they are scattered in the ROM and other times they can even repeat.

<POINTER4> random data size <POINTER2> more data <POINTER1> more more more more <POINTER1>

In this case it's easiest to dump the script as raw and then go back and find the pointers after. However, a pointer scanner may not be totally accurate but it would be able to make some of this task automated.

So a script may look like this

Code:

#W16($0000)
A text string here.<end>

//Pointer not found
Another string here.<end>

Dragon Quest 5 for ps2 hacker/translator....&&

IP Logged

Nightcrawler

YaBB Administrator

Offline

The Dark Angel of Romhacking

Posts: 3251
USA
Gender: male

Re: GUI Based Generic Script Dumper
Reply #5 - Jul 13^th, 2010 at 11:28am

What algorithm do you have in mind that would scan and be able to find pointers reliably?

The thing is they're not really scattered and the spacing isn't random. I think it would be better to know what you're dealing with. Are they hard coded? Are they in a scripting block?

If so, I'd probably suggest a pointer list for this rather than a faulty scanning mechanism that gives you a guess. Then you can order and rearrange things. I believe Atlas has this ability. I would certainly consider support for an external pointer list.

Now, I've done things where a scripting block is scanned for string commands, or a ROM chunk is searched for opcodes resembling hard coded pointers. In both cases, my results were certainly unreliable and required much manual investigation in false positives or even missed occasions.

What you're suggesting might work better as a separate scanning utility that generates a pointer list to be used for dumping. It could be rule based or have a few different scanning algorithms etc. Maybe even a relative searcher using pointer size and endian searching for increasing values. There's a few things you could do. Though it's kind of like using a corrupter. Yes, it can work, but it's probably better to use techniques that will gives you a concrete and reliable answer.

I have a hard time thinking of an algorithm that would work across different platforms/consoles and pointer size/endian.

ROMhacking.net - The central hub of the ROM hacking community.

IP Logged

Gil Galad

Peasant

Offline

I Love TransCorp!

Posts: 5
Ohio, USA
Gender: male

Re: GUI Based Generic Script Dumper
Reply #6 - Jul 14^th, 2010 at 6:59am

I really think you're going in the right direction with this script dumper. It looks very promising and hopefully you'll be updating it upon occasion.

As far as out of order pointers are concerned, I don't think that will be a problem at all. When you input the range of the pointer table, it simply dumps them to a text file and if you have the script Atlas compatible, the address of the pointer will be written under the small block of text.

I used Cartographer for Star Wars (Namco) and it had some out of order pointers which were handled easily.

One thing about Cartographer that's annoying is that it outputs #W16 for each string that's referenced by a pointer. So, you have to go to each one and change it to whatever label you need. For a really large script, this is a problem and time consuming. I talked to RedComet about adding better support but I don't think he has the time for it currently. I think that Cartographer is a great concept and it works well with a few exceptions.

I think that there should be some type of input box in the dumper somewhere where you can add custom pointer writes. When you dump just output in the text dump file. Instead of doing #W16 for each one, for example.

Here is a small example of how it could look like.

Code:

#SMA("LINEAR")
#HDR($10)
#VAR(dialogue,TABLE)
#ADDTBL("mad_ea.tbl", dialogue)

#VAR(motto, CUSTOMPOINTER)
#CREATEPTR(motto8000, "LINEAR", $-8000, 16)

#ACTIVETBL(dialogue)


//*********************
//**START OF 12842.txt**
//*********************
//GAME NAME: The Most Dangerous Detectives
//BLOCK #000 NAME:		Dialogue Block (POINTER_RELATIVE)

#JMP($12842,$1300E)

//Displayed - Intro dialog
//POINTER #0 @ $12810 - STRING #0 @ $12842
//タカ  「きのうのおんなはどうした。ユウジ 「ごぜんれいじの チャイムと     どうじに わかれたさ。[SECT]
//タカ  「いいおんなだったじゃねえか。ユウジ 「せいかつを     かえろっていうんだ。[SECT]
//タカ  「おとこを     かえようとするもんさ。     おんなは。[SECT]
//ユウジ 「かえられるか いまさら。タカ  「むりだろうな。[SECT]
//ユウジ 「こんな おもしろいしごと     やめられるか。[END]
#WRITE(motto8000, $12810)

I'm also really tired right now, so if I'm not clear about this, I apologize.

IP Logged

Nightcrawler YaBB Administrator Offline The Dark Angel of Romhacking Posts: 3251 USA Gender:	Re: GUI Based Generic Script Dumper Reply #7 - Jul 14^th, 2010 at 11:01am What would the idea be there? To reorder the strings for better organization, but have the pointers remain the same? The issue with the #W16 from Cartographer is what exactly? You have to manually change it to 'whatever label you need'?
Back to top	ROMhacking.net - The central hub of the ROM hacking community. IP Logged

Nightcrawler

YaBB Administrator

Offline

The Dark Angel of Romhacking

Posts: 3251
USA
Gender: male

Re: GUI Based Generic Script Dumper
Reply #8 - Aug 2^nd, 2010 at 2:05pm

I'm thinking of calling this 'TransText' or 'TextAngel'. I'm not sure yet. Any opinions?

I've been adding more features and testing things out since the last update. I ended up revamping the interface to something that would scale a little better with the new features I was trying to add. I've gone to a tab based interface as you can see here:

http://transcorp.parodius.com/scratchpad/TextAngel.png

I added support for pascal style strings and re-dumped the Herakles IV script to test using Pascal Strings and the new Table File Standard. It's coming out looking nice.

I'm still trying to work on refining/polishing the interface. I'm not even sure how I'm going to handle situations where you want to dump say a directory of binary decompressed files. Or cases where you have a master block of absolute pointers to individual text blocks with their own pointers. Things like that I find difficult to figure out an interface for. I welcome to suggestions.

I've also added a few debugging features that I have yet to make optional.

It's certainly a work in progress still, but it's becoming usable. I'll probably make some more screenshots when I stop changing my mind on things!

ROMhacking.net - The central hub of the ROM hacking community.

IP Logged

Neil Knight of the Kingdom Offline In the Lost and Found Posts: 315 Vermont Gender:	Re: Text Angel (GUI Based Generic Script Dumper) Reply #9 - Oct 30^th, 2010 at 1:13pm TextAble? Although it would be nice if you went with a term that fits with the Cartographer/Atlas schema, especially if you're output is something reinsertable by Atlas. Annotator? Doesn't really have a clever ring to it, but maybe worth consideration.
Back to top	IP Logged

Nightcrawler

YaBB Administrator

Offline

The Dark Angel of Romhacking

Posts: 3251
USA
Gender: male

Re: Text Angel (GUI Generic Script Dumper/Inserter)
Reply #10 - Apr 7^th, 2011 at 1:50pm

Check out the first post for several new images and explanation of how things are currently working. I didn't post here for awhile because I just kept spinning my wheels changing the interface over and over. I'm still not entirely happy with it, but it seems to be the best I can think of. It got especially tricky when I decided I want it to be able to insert what it can dump as well. That's when I thought of a unified interface. You basically need the same information whether you're dumping or inserting, so why can't they share the same interface in the same utility? If nothing else it's good thinking fodder!

ROMhacking.net - The central hub of the ROM hacking community.

IP Logged

Nightcrawler

YaBB Administrator

Offline

The Dark Angel of Romhacking

Posts: 3251
USA
Gender: male

Re: Text Angel (GUI Generic Script Dumper/Inserter)
Reply #11 - Jun 27^th, 2011 at 11:12am

For those of you checking here for news about this, I was seeking design advice from some guys over at RHDN:
http://www.romhacking.net/forum/index.php/topic,12462.0.html

This gave birth to adding some feature for more advanced usage:
http://transcorp.parodius.com/scratchpad/interesting.png

I am getting there. It's shaping up to be something nice. I will hopefully soon pull together a little write-up on all the features TextAngel supports. Unfortunately, it's still been in a constant state of flux as it morphs and twists it's way into what it needs to be. Aside from being based on an emerging and changing table file standard, my initial design and outline just didn't cover everything. All kinds of new business and missed cases started coming out. Then, my novel idea of dumping and inserting with the same interface became a bit more difficult than I thought as the features supported became more complex. I may be a seasoned ROM hacking veteran, but I still have a long road ahead in good software design. It's been a silly mix. I know precisely what I want and need to do conceptually as far as ROM hacking application. Then I hit a bunch of walls when my program design fails to scale, my user interface comes out awkward, or I'm just not sure how provide for certain ROM hacking cases. Smart guy on the left hand, dumb guy on the right hand, and they have to get together, shake hands, and come up with something good! Sometimes it makes me laugh! Grin

Like everything I do these days, the pace is slow, but the fruit will be worth it when it's completed.

ROMhacking.net - The central hub of the ROM hacking community.

IP Logged

Next_Gen_Cowboy

Nobleman

Offline

I am what I am; nothing
more and nothing less

Posts: 113
Inside my own head.
Gender: male

Re: Text Angel (GUI Generic Script Dumper/Inserter)
Reply #12 - Jul 18^th, 2011 at 11:24am

It's interesting to see, and I think the basic concept as a whole applies in almost every case. Unexpected problems, a user-base that varies in talent, and skill. Seeing new possible aplications and implementing them, desiging a UI; I think it all just comes with the territory.

The important aspect at least you seem to have covered. You understand what you need, and what it needs to do. Interfaces suck anyway Tongue

They're just another thing I could never get right, or make work no matter how I went about it.

Sleep is like cocaine, for the brain.

IP Logged

Gil Galad Peasant Offline I Love TransCorp! Posts: 5 Ohio, USA Gender:	Re: Text Angel (GUI Generic Script Dumper/Inserter) Reply #13 - Jul 26^th, 2011 at 7:24am If Text Angel is at a working state right now, I could possibly test it on a small NES game like I did with Cartographer a couple of years ago. Up to you, of course.
Back to top	IP Logged

Nightcrawler

YaBB Administrator

Offline

The Dark Angel of Romhacking

Posts: 3251
USA
Gender: male

Re: Text Angel (GUI Generic Script Dumper/Inserter)
Reply #14 - Jul 28^th, 2011 at 4:00pm

TextAngel has seen huge progress over the past several weeks! It's in a generally working state and will need much testing. However, it's based on the still not finalized new table standard. As a result, there are several issues with Atlas compatibility with the current version due to some generally incompatible operations, as well as some table standard differences.

TextAngel strives to not only be Atlas compatible, it strives to be fully 'Atlus Ready'. What I mean by that is it will output all necessary Atlas commands for most of it's script dumps. It will theoretically be ready to re-insert out of the box if you didn't want to make any location changes. It will easily do what Gil Galad spoke about earlier and use CUSTOMPOINTERS and figure out when using AUTOWRITE is appropriate. I am having some difficulty as many things in TextAngel don't have 1:1 translations in Atlas, and there are incompatibles as TextAngel uses the new table standard and Atlas does not at this time. Also some things are just very difficult to automate Atlas output on, especially when some Atlas quirks are not fully understood by me. Wink

I've also just finished ironing out the AutoDiscovery/Pointer Scan feature where given a text block, it will dump strings and scan/discover pointers in the pointer area based on the specified format similar to what DaMarsMan said earlier in this topic.

I also added a method to do a dictionary dump as well as re-order by string location and duplicate string check this week! I would say it's probably the most advanced public dumper ever.

Here are some of it's many features:

Batch File Support - Automated dumping across many files.
Table Standard 1.0 WIP draft support.
Raw Start/Stop Dumps
Pointer Table based Dumps (Also can ignore invalid pointers if desired)
Pointer List based Dumps for scattered pointers.
LOROM00, LOROM80, HIROM, GB, LINEAR modes
8-bit, 16-bit, 24-bit, 32-bit pointer support
Little and Big Endian
Pointer Spacing (Spacing between Pointers)
Pointer per byte spacing (arbitrary Spacing between individual bytes in a pointer)
Relative Pointers with Table Start as Base
Relative Pointers with Pointer Location as Base.
Pointer Global Offset
Header support for pointer calculations.
AutoDiscovery/Pointer Scan based dumps
Atlas pointer compatibility
Full Atlas Ready output out of the box. (Currently having issues as this can complicated for some of the complicated features and structures TextAngel supports.)
EndTerminated Strings
Pascal Strings
Fixed Length Strings
Fixed Length Strings Based on Pointer Calculations (size=pointer(n+1) - Pointer (n))
Fixed Length Lines
Dictionary Dump (Uses string prefix/postfix ability)
Re-ordering dump by string location
Duplicate string detection
Overlapping Strings detection
XML Export
Dump Templates
Advanced Tree view with dump preview
UTF-8, S-JIS, ASCII, and JP-EUC encoding support for tables and script dump.
Project support for dumping all areas of your project. (script, items, menu etc.)

By the way, TextAngel DOES insert text as well, but it's not nearly as developed in that area yet. It supports all of the same general features for insertion as it does for dumping, but much of the nuts and bolts are not yet implemented there. It does insert the Heracles IV script though.

« Last Edit: Jul 29^th, 2011 at 2:31pm by Nightcrawler »

ROMhacking.net - The central hub of the ROM hacking community.

IP Logged

Pages: 1 2

Nightcrawler's Message Board › Rom Hacking/Translation Board › Rom Hacking/Translation Board

‹ No Topic | Next Topic ›

(Moderator: Nightcrawler)