2020-11-18

Forgive if it does you good, but forget at your own peril

The Election is Settled

The 2020 elections are over here in the USA, and though it took longer than usual to project a winner in the presidential race, with all the major news outlets seemingly afraid of being the first to call Pennsylvania, there can no longer be any plausible doubt about the election's outcome. With a final electoral count expected to match Donald Trump's official victory margin in 2016 (not to mention the most votes in history for a candidate, resulting in a popular vote victory margin of over five million), Joe Biden is rightfully president elect. This was largely settled well over a week ago and has only become more certain since.

But still the fighting continues

Yet even as of this writing (and, unfortunately, as many expected) Donald Trump refuses to concede, instead launching a volley of desperate ill-conceived lawsuits and continuing to promote increasingly baseless claims that the entire process was heavily and illegally biased against him. In reality the gutting of the Voting Rights Act along with widespread gerrymandering, removal of voting places, voter deregistration (particularly in states like Georgia), misinformation, obstruction of mail-in voting during the worst pandemic in a century, and multiple other forms of vote suppression by the GOP mean it's far more likely that the opposite is true.

Many government agencies are following his dubious lead and refusing to cooperate with the incoming administration, to the detriment of the nation. Prominent Republican leaders, such as professional obstructionist and court stacker somehow-still-a-Senator Mitch McConnell, also refuse to accept the results. As of November 16, only five Republican Senators have publicly acknowledged Biden as president-elect. In many cases, they're carrying on as though a Trump victory were a done deal, citing allegations not borne out by evidence, if they give any reason at all. Pressure has been put on Republican-controlled state legislatures to throw out the results of their elections and appoint pro-Trump electors (though thankfully few, if any, seem inclined to consider doing so), and similarly on the Supreme Court to find some pretext to award Trump a win. Secretary of State Mike Pompeo went as far as to promise "a smooth transition to a second Trump administration" days after the results were known. And don't even get me started on the conspiratorial nonsense and worse floating around in right wing media and the darker corners of the Internet.

Despite these ongoing attacks on truth, decency, and the law, I keep seeing calls for Trump opponents to forgive and forget, to let it all go, to empathize with Trump supporters. Even many leaders in the Democratic party are urging a move toward the center, blaming "extreme" leftist positions for losing down-ballot races, despite a lack of any evidence to support this and despite the popularity of many of these positions when not specifically framed as leftist. It seems like no matter what happens, it's the left and liberals and Democrats (which are not interchangeable terms) who are always called on to move toward the center.

But what is the center?

In many ways, this reminds me of the United Methodist Church's disastrous politicized arguments over how to respond to the existence of gay people.

In short, the denomination's rule book has some problematic outdated homophobic language that had been enforced only inconsistently of late, and there's been a long-running dispute over what to do about it. Two plans emerged as likely possibilities for future policy. The One Church Plan would remove the offending language and leave it up to individual churches what to do in their own congregations from there. The Traditional Plan, opposed by the commission entrusted with planning a way forward, would not only keep it but emphasize enforcing consequences on any church or clergy that dared step out of line.

Our church's pastor, a devoted disciple of the Golden Mean fallacy, cast it as a heated dispute between two extremes that wouldn't listen to each other. Yet that view doesn't accurately reflect the reality of the situation. It hinges on the assumption, not only that neither plan was actively harmful, but that the One Church Plan and the Traditional Plan were largely equal and opposite. This is patently false.

The One Church Plan was an attempted compromise that emphasized unity at the expense of a disproportionately vulnerable and long-maligned population. An appropriate counterpart to that would have emphasized human dignity even if it came at the expense of unity, a position further away from the Traditional Plan than the One Church Plan itself. A meaningful counterpart to the Traditional Plan, on the other hand, would have been far more dramatic, sanctioning and even expelling churches willing to misuse the name of Christ to promote or condone bigotry.

The choice presented was never between a liberal option and a conservative option, but a relatively moderate conservative option and a completely unreasonable punitive option. The middle was never between them to begin with, but beyond them. The center was never once seriously considered as a workable option. And, aside from the effects on those put forth as pawns, the whole argument may have been little more than a proxy for a deeper power struggle to begin with.

Regardless, when it came to a vote, the extreme prevailed and the relatively moderate was spurned. And because that didn't result in everyone making up and singing Kumbaya, the pastor told us, along with other painfully tone-deaf remarks, that "nothing has changed". Never mind that the church had just effectively said, "you're not welcome here, and neither is anyone who cares about your well-being."

Coming as it did as part of an ongoing pattern of framing nearly anything and everything as a matter of two equally valid sides that just need to try harder to get along (something I'm likely to vent about in more detail eventually), that particular bad take was, for me, the last straw. Did nothing register beyond whether or not there was still vocal dissent? Was nothing more important than "open conflict bad (no matter the reasons), ostensible peace good (no matter how superficial)"? Or maybe it ultimately came down to a personal blind spot, a simple preference for the familiar and comfortable, unburdened by the complexities of reality. It was difficult to tell, and my willingness to keep extending the benefit of the doubt or remain any longer in such an unhealthy situation had been exhausted.

The failure of "both sides" framing

Something similar has been happening in American political discourse. The same pastor I find myself so disappointed in (I'm still monitoring the weekly newsletter in the admittedly vain hope that something meaningful will change) called the election results a triumph of the center over "radical fringes", even though most Democrats are, by any meaningful standard, at most only slightly left of center, while most of the Republican party has largely gone off an ideological cliff to the point of openly pandering to racism and other bigotries, and even running dozens of outright conspiracy theorists for public office across the nation. Several of whom won election. The center didn't triumph; it's struggling to survive.

I often find that statements dismissing "both sides" as problematic, when not simply an effort to excuse bad behavior on one side by claiming another side is just as bad, say less about either of these sides than about the speaker. It often seems connected to a desire to think oneself better than those who take a side, and furthermore, to dismiss rather than engaging with the concerns of anyone who holds a meaningful position on any divisive issue. At best, it's lazy, making little or no effort to examine any side in any detail, failing to recognize that there often are more than just two sides, and furthermore ignoring that seldom are any two sides ever culpable in the same way or to the same extent.

The media has a bad habit of doing much the same thing. Anything Donald Trump and his lieutenants say is treated as inherently newsworthy and often uncritically repeated with no attempt to note its relevance, must less its accuracy, which just amplifies bad-faith propaganda. Glaring abuses of power are often dubbed "unconventional", seldom "illegal" even when they flagrantly are. Conservatism as a whole is consistently given the benefit of the doubt and graded on a curve, ironically creating actual bias in a misguided effort to avoid the appearance of bias. And the media largely remains invested in the idea that we still have two equally valid political parties, both essential to the proper functioning of the nation, even as one has become increasingly anti-democracy, increasingly dependent on abusing the system to maintain power, increasingly willing to follow the rules only when convenient, and increasingly beholden to the alt-right.

Social media just makes things worse. Though conservatives love to accuse platforms of anti-conservative bias, study after study has found that politically right-leaning pages consistently outperform left-leaning ones on Facebook, which reportedly made a deliberate decision to signal boost low-quality right-wing sources like Breitbart at the expense of more liberal higher-quality ones. Twitter has become notorious for letting Trump in particular repeatedly post misinformation and inflammatory content that would get anyone else banned, with at most an ineffective warning label (which, fascistic outrage aside, is in no way censorship and plainly hasn't silenced anyone). Not only that, but they have reportedly declined to crack down on neo-Nazi and white supremacist content rather than dealing with the outrage of the Republican politicians whom any such effort would inevitably impact. And it's probably best not to even think about specifically right-wing platforms, like Gab and Parler, that function to shield from disagreement those who can't abide opposing viewpoints.

Meanwhile, the messaging coming from right wing sources overwhelmingly paints those who don't agree with them, or who don't fit into their idea of how the world is supposed to work, as not merely the enemy, but in many cases as outright evil or subhuman. The specific target chosen matters less than having someone to blame. Be these liberals, leftists, socialists (which seems to mean anyone who supports any kind of safety nets other than corporate bailouts, or who has any reservations about unbridled crony capitalism), immigrants (unless from approved, sufficiently white, sources), feminists, women in general unless they know their place, sex workers, Jews, Muslims, atheists, and even Christians if they're not the "right" type, scientists, the educated in general, millennials, the poor, "elites" (whoever those are; it's never clearly defined), sexual minorities, transgender people, anyone who doesn't adequately conform to obsolete gender norms... The list goes on and on, and we're all fair game for demonizing and scapegoating, especially those groups of people that have the least ability to meaningfully push back.

White evangelical churches are some of the worst offenders when it comes to "othering" people, habitually claiming religious freedom as an excuse to do so. Not to mention crying "oppression" at any attempt to push back against allowing them to impose their particular tenets of morality on everyone else. All while numerous faith leaders keep claiming to speak in the name of God—and cynically making bank on doing so—when they make baldly political prophesies that have already been falsified, or when they denigrate as ruinous and Satanic anything and anyone that doesn't hew to their narrow, self-serving view of Christianity. Can I get a "woe to you, hypocrites"?

Finding a vulnerable group to "other" has a long history when it comes to promoting reactionary causes, though. From centuries-old blood libel to decades-old Satanic Panic to the largely incoherent and often self-debunking QAnon conspiracy theories ascendant today, "Satanic baby-killer" rhetoric paints the other side as so cartoonishly evil that it's easy to justify nearly anything to oppose them. If they get hurt in the process, well, they were obviously asking for it by being Satanic baby-killers. And if innocents get caught in the crossfire, well, as Lord Farquaad remarked in Shrek, "Some of you may die, but it's a sacrifice I am willing to make." It's all in the name of opposing Satanic baby-killers, after all.

This rhetoric has consequences, and not just in the workings of government. Take Heather Heyer, fatally run down by a white supremacist. Or the mass shooting at the Tree of Life synagogue in Pittsburgh. Or several incidents involving heavily armed right-wing protesters terrorists forcing their way into government buildings, which in at least one case resulted in canceling a legislative session. Or the teenage killer who made a trip to a city in another state for the chance to shoot at those protesting racism and police brutality. Or the militant group who plotted to kidnap and publicly execute Michigan Governor Gretchen Whitmer and set fire to the capitol. Or the extremist groups like the Proud Boys even now rioting against the election, supporting fascism over democracy. Just to name a few.

This isn't coming from nowhere. And even when the worst happens, it too often gets the "both sides" treatment. All while a significant portion of the right wing defends the terrorists—the Kenosha killer is even being lauded as some kind of folk hero now—and keeps claiming to be the party of law and order!

One side largely consists of people who just want to live their lives in peace, without being unduly harassed, threatened, or maligned for existing. The other side bases a large part of its messaging on despising and denying who these people are, spends a large part of its political clout on curtailing their rights and making it more difficult for them to exist, and furthermore depends on doing so to maintain a significant portion of its voter base. Which results in things like "Fox & Friends" calling Mister Rogers, of all people, an "evil, evil man" while misrepresenting the values he tried to teach.

Considering that it's currently Transgender Awareness Week, let's take a moment to point out a few lowlights of attacks on our civil rights in particular. Like the time Trump banned transgender people from the military (via Twitter!) for no substantive reason. Or the efforts in multiple states to ban transgender students from participating in school sports, while largely neglecting the uncontrolled pandemic that means school sports aren't safe for any student in the first place. Or supposed leaders who not only reject calls to ban the abusive and ineffective practice of conversion therapy, but argue that keeping people out of it is the real abuse. Or how the White House is even now fighting to allow health care providers the special privilege to refuse to treat people who don't look cisgender enough to calm their prejudices. These aren't the actions of people attempting to address real issues. They're categorizing people as an out-group, then punishing them for being classed in that group. They're pandering to bigotry in exchange for political backing.

Yet the right wing rarely seems to be called on to moderate its positions or actions, or even its rhetoric, and it feels like Democrats (not to mention the news media and social media) keep giving in to Republicans on practically everything, resulting in what I've seen compared to a political version of Zeno's Dichotomy Paradox. If one side keeps meeting the other halfway while the other doesn't budge (or worse, grows ever more extreme), that just keeps creating new "middle grounds" that continually shift further and further away from anything resembling a center, whether the two sides ever quite meet or not.

It's been said that the core tenet of conservatism has always been that there must be in-groups that the law protects without constraining, alongside out-groups that the law constrains without protecting. It's been said that reactionaries are largely incapable of imagining other people thinking differently from themselves, making every accusation they level against their opponents a confession of what they would do or have actually done. It's been said that the only true principle of fascism is domination, and that hypocrisy is therefore a virtue to a fascist, for what clearer way is there to exercise power over others than to indulge in what you deny to them? There may be more to the American right wing than these, but if so, recent years certainly haven't shown it.

And though the voters have spoken, we still have more than a month left of a Trump administration with an obliging Senate and Republican-dominated courts. This hasn't been a stable, responsible, or reasonable administration even at the best of times. It's likely to get worse before it gets better. Already, Trump has been firing officials haphazardly, and there are reports of infighting.

On Being The Reasonable One

On that note, all the gaslighting, domineering, and everything else brings to mind an abusive relationship. A concept that keeps coming up in Captain Awkward posts involving (usually) families with abusive dynamics is The Reasonable One. When you're The Reasonable One in the family, everyone always expects you to give in and make nice and let yourself be walked all over, because, whether they're willing to admit it or not, they know who the unreasonable one is and that it's pointless to try reasoning with them. And so, when there's conflict, the pressure always falls on you to fix things, even though you're not the one responsible. The one who is responsible isn't expected to do anything; after all, everyone knows better than to expect anything from them.

One of my favorite responses for this type of situation involves unambiguously naming bad behavior. Don't dance around it, but bluntly call it what it is. For example, if you have an extended family member who always pressures younger female relatives into inappropriate hugs and has repeatedly held a burning lighter against your sister's arm, don't let people wave it off as harmless or talk about it in euphemisms. They're creepy uncomfortable hugs, at best, whether they're meant to be or not. It's assault with a cigarette lighter, not innocuous "playing around". If family members ask why you're ruining Christmas (or whatever), ask them why they keep letting him ruin it for you.

The specifics depend on the circumstances, but overall, dealing with a problem where being The Reasonable One is unacceptably harming you mainly comes down to one thing. There comes a point when the only reasonable response is to stop being The Reasonable One, at the very least until some semblance of fairness is restored. Make it clear where the problems are coming from, and that you refuse to accept responsibility for other people's actions. If they want to have a harmonious relationship—or possibly any relationship—with you, they need to do something about their misbehavior and how it's harming you.

Survivors aren't responsible for mending bridges with their abusers, and placing that responsibility on them is itself abusive. Forgiveness doesn't fix much of anything if there's no corresponding repentance.

Sometimes loving your friends and family means calling them out. Sometimes loving your enemies means doing what you can to keep them from harming others, and, yes, themselves as well. Sometimes loving yourself means avoiding people you can't trust not to threaten your well-being. None of these mean pretending that there isn't a problem when there most certainly is one.

To name just one bad behavior in the current political climate, undermining healthcare, spreading misinformation, and disparaging basic safety measures during the worst pandemic in a century isn't a simple difference of opinion. It's killing hundreds of thousands and creating lasting health problems for countless more, all while crushing the supposedly all-important economy by making it simply not safe for us to go about our business normally.

Anyone willing to support the most criminal, most corrupt, and arguably the most bigoted and uncaring administration in the history of the nation shouldn't be surprised if people are reluctant to associate with them. Even if for non-fascist reasons, supporting fascistic actions still supports fascism. Even if not out of racism, supporting white supremacists still supports white supremacy. This can't be waved off as just someone's harmless difference of opinion, or as "it's just politics" as though it didn't reflect on their values or have any real-world impact.

The Trump administration has spent the last four years (and far longer, for many of its individual members) oppressing the marginalized, playing favorites with those willing to kiss up to them, undermining the rule of law, attacking journalism specifically and the very concept of truth in general, and encouraging others to do the same and worse. The Republican Senate has at best allowed this to happen and at worst assisted in it. And the media, perhaps in a futile attempt to ward off criticism from those who refuse to play by the rules regardless, has kept framing flagrant lawlessness as routine controversy.

Even aside from the myriad specific harms they've perpetrated in just four years, Trump and his enablers—and it's difficult not to count the entire Republican party among them—have spent the whole time making things up and lying at every turn about even inconsequential matters, when not outright denying undeniable realities. None of this is normal. It cannot be worked with. It must not be accepted. And it needs to change before there's any possibility of reconciliation.

It hasn't yet and shows no signs of doing so.

What are The Reasonable Ones expected to do?

The summary above of a few of the recent actions those in power have taken against transgender people is just part of a much longer list, and only covers one type of minority. Others have been treated as poorly, and worse. See how the recent protests against racism and police brutality have been met with even more racism and police brutality, for just one example. How can you exist in harmony with someone who doesn't believe you should have the same rights as everyone else, or even exist?

Neil Gorsuch was added to the Supreme Court only after the Republican Senate unilaterally reduced it to eight seats for over eight months (and similarly kept hundreds of seats on lower courts empty), with threats to keep it that way for longer and reduce it further if Hillary Clinton became president. Brett Kavanaugh was installed almost without a hearing despite numerous serious questions about his suitability, honesty, temperament, finances, and loyalties, on top of the rapey behavior that got all the attention. Amy Coney Barret (only a judge to begin with thanks to one of those lower court seats illicitly kept empty for over a year) was rushed through confirmation despite lacking relevant experience and having a history that promises embracing a particular niche brand of theism. Samuel Alito recently broke with the tradition of maintaining at least a semblance of impartiality by giving a bitterly partisan grievance-ridden speech that rightfully ought to disqualify him from hearing cases on just about anything. When the Supreme Court, supposed to be objective and unbiased, is so loaded with problematic members and so openly tainted by partisan politics, how are we expected to put any faith in it?

One QAnon adherent who was somehow elected to the House of Representatives, newly arrived in Washington D.C. where businesses currently remain open to the public, recently complained spuriously on Twitter about seeing closed businesses everywhere and not being able to go to a gym for CrossFit. Locals quickly pointed out that not only is none of this true, there's literally a gym right around the corner from her hotel offering CrossFit, right now. She responded by crying "fake news" and doubling down on her counterfactual claims. What meaningful cooperation can you have with someone who rejects shared reality and substitutes their own? And how has caring what the truth is become such a partisan issue?

There are people now (as there were a century ago) avoiding basic health and safety precautions as not "manly" enough for them, if they even acknowledge the pandemic in the first place. No small number of conservatives have taken the general concept of ideology over practicality further still, embracing an attitude of "owning the libs" even to their own detriment. Some evidently base their entire identity on being belligerently anti-progressive in a way that makes it seem like cruelty is not just acceptable fallout but the whole point. "Fuck your feelings" is an unofficial slogan of the right wing (not their feelings, though; those, apparently, are paramount). How has giving a damn about other people become such a partisan issue? What possible way is there to reason with people operating out of sheer spite?

I won't advocate simply giving up on people, assuming they're willing to listen. But many aren't, while myriad harms continue unabated. The only apology that means anything is changed behavior. There needs to be something to work with, some assurance that the same thing won't keep happening all over again, and there's been all too little sign of that.

Wishful thinking only goes so far. It takes two to cooperate. If one outright refuses to, what then?

2020-06-04

The police are rioting

And it's happening in cities across the nation. If they could maybe, you know, not? And also refrain from killing (mostly black) people for funsies? And perhaps deign to allow some degree of accountability that it's increasingly obvious is lacking at present? And just generally stop confusing themselves with soldiers at war minus Geneva Convention rules? That'd be great.

2020-03-31

TDoV 2020

(The following was written for presentation as part of a local Unitarian Universalist service celebrating the 2020 Transgender Day of Visibility. The opening directly quotes from a previous blog entry, so should look familiar to anyone who's read that.)


It's been about two years, now, since one branch of a major Christian church justified its anti-transgender stance with an official statement bearing a name that referenced Genesis 1:27, as though this were not only an obvious and natural conclusion from the text of the verse, but the only conclusion possible.

As the verse reads in the New International Version: "So God created mankind in his own image, in the image of God he created them; male and female he created them."

"And" should be such a simple word.

Particularly for something written in a time when women were often considered little, if at all, better than property, that verse from Genesis makes a fascinating assertion. The image of God is not the sole domain of the male, nor, for that matter, of the female. Neither masculine nor feminine is better or more Godly. The man cannot say to the woman that he is more favored of God, nor the woman say to the man that nothing of God is in him. All are part of God's creation and exhibit facets of God's nature.

Created "in the image of God... male and female". Not only does this put both on the same level, it seems an unavoidable conclusion that the image of God is therefore neither purely male nor purely female, that this is a God who is both and neither. Why should those created in that image have to be purely one or the other?

But that isn't what not only religious groups, but also culture and media have for many years shown us. Schoolchildren are divided up into boys on one side, girls on the other. A popular book asserts that "Men Are from Mars, Women Are from Venus". Facilities are labeled "ladies" or "gentlemen". Paperwork often demands an "M" or an "F", even when it's completely irrelevant. Stores keep women's clothes over here and men's clothes over there, as though they were intrinsically incompatible. Seldom is there any hint that these might not be rigid, eternal, unchangeable certainties, much less that there could be anything between or beyond them.

So perhaps it's not so strange that I can remember being in high school with a fuzzy not-really-understanding that there were "normal" people over here, and "the gays" over there. I knew I liked girls too much to be one of "the gays" (which was just about all I knew about what that even meant), and even if boys sometimes sparked my interest, too, I also had a vague cultural-awareness sort of sense that "the gay" was best avoided. So I mostly tried not to think about the idea that I might not be "normal", either, because what else was there? And If I developed a certain fascination with things like unisex names, or fiction that involved people magically changing sex, or wondering how my life would be different if I had been born a girl, surely it didn't mean anything in particular, but was nothing more than idle curiosity from a mind bored with the tedium of schoolwork.

The media has generally gotten somewhat better since then, but even now, a lot of "representation" is still stuck on the idea that if you're not toeing the gender line, then you're either dead or working the street corners or trying to trick people for malicious reasons, or, at best, you're a punchline. Even attempts to be more respectful and speak to transgender people as people have an unfortunate tendency to get uncomfortably focused on the state of people's genitalia—which, to paraphrase John Oliver, is, medically speaking, none of anyone else's business. And, as with so many other topics, journalists are often so eager to avoid taking sides that they'll put verifiable facts, legitimate scientific studies, and real people's safety and wellbeing on the same footing as fanciful speculation, extremist propaganda, and baseless fearmongering.

The Internet, although it can amplify the worst voices, also allows others to be heard. For instance, I'm reading a webcomic by a trans author with characters that include a trans girl who's gone full time after moving and changing schools, a trans man who is her therapist, at least one lesbian, several people who are bisexual, a gay boy who likes to wear dresses and be pretty now and then, one girl who's pansexual, another who's described as asexual and homoromantic, someone else who's genderfluid and usually uses they/them pronouns, an intersex girl who was raised as a boy, a trans boy who's just starting to figure that out, a boy who might be gay or bi and is both curious and terrified to find out, and various authority figures and family members who run the gamut from supportive to confused to indifferent to openly hostile. Another webcomic, in a more sci-fi/fantasy genre involving transformation technology and magic, includes characters such as one with form-changing spells who strikes me as bigender and recently used the description "gender-casual", a female duplicate of an originally male character who has described herself as homoromantic and bisexual, a part-alien shapeshifter who's most likely demisexual and agender, a transformation enthusiast who has been confirmed as genderfluid, and someone who finds people of all genders attractive and appreciates sexuality between other people but is repulsed by the idea of being personally involved—and is there even a word for that? (ETA: of course there is)

But you're not likely to find nearly so much visible diversity unless you deliberately seek it out. Most of these concepts still aren't really out there in the public consciousness, much less taken for granted. And, ideally, they should be, if for no other reason than making sure that everyone knows they don't have to try to force themselves into this box or that box. There's no single standard that everyone has to live up to, and no preset path that everyone has to follow. There's just people being who they are as well as they can manage in the circumstances available to them.

2020-03-29

Text Encoding and Compression (featuring Lufia 2)

One of the things I have a lot of on my website is attempted translations of video game text, mostly SNES-era RPGs since that's what I grew up on and have the most familiarity with. This started out as something I did out of my own personal curiosity and for practice, before it occurred to me that I might as well go ahead and share it. Here's a little more insight into part of the process.

Text dumping overview


If you're interested in translating the text of a game, the first thing you need is, of course, that text. Naturally, you could transcribe it from a playthrough of the game, reading it off the screen and typing it manually, but that's both time-consuming and error-prone. It also has the drawback that you need to get the text to display in-game before you can do anything with it, which is problematic in scenes with multiple branches and variants, and means you may never even know about easily-missed text, let alone text that was left entirely unused.

Ideally, what you want is to pull the text directly from the game data. One of the biggest problems with that, assuming you have access to the game's raw data in the first place, is that the text is often encoded in some nonstandard fashion, and usually also compressed on top of that. This is particularly true of cartridge-era console games, which had to be developed within strict storage limits. All that often makes it difficult to even locate the text, much less extract it in an intelligible format.

Lufia II has been probably the most interesting case I've encountered so far. This game's text encoding (in the English version, at least) is based on standard ASCII, a rarity among console games of its time. That sounds like it ought to make picking out the text fairly easy. And in some places, it does! Spell, item, and monster names, for instance, are entirely readable without any additional interpretation, though there's sometimes some unreadable stuff between names. For instance, part of the item list follows (they're fixed-length entries; line breaks are added here for readability):

Charred newt
Potion      
Hi-Potion   
Ex-Potion   
Magic jar   
Hi-Magic    
Ex-Magic    
Regain      
Miracle     
Antidote    

However, most of the dialog looks more like this (unprintable bytes are shown as their hex values in braces, using the convention of a dollar sign prefix):

Y{$11}3{$01}{$00}1{$04}{$04}7{$0A}fI{$8E}{$06}5 {$05}y ok{$0E}{$09}? {$05}{ {$8C}Ðt{$03} {$81}Ì·ð{$01} aI{$9E}{$06}@ {$06}{$97}{$8E}¸ {$06}è{$03} {$06}N {$0A}É"y{$84}{$06}« {$8B}{$0E}{$05}{$7F} {$06}Ì.{$01}

So what went wrong? As I mentioned above, the encoding is only based on ASCII. Quite a few byte values have special meanings (for instance, $09 in a text block stands for the protagonist's name). More importantly, the text is compressed to save space. Not only that, this game uses three different types of text compression.

Basic text compression


Let's start with the simplest one. Numerous byte values, including everything from $80 through $FF, stand for common combinations of (usually) two characters. (This only applies to the English version. The Japanese version uses single-byte codes for each of 162 kana, plus 52 upper- and lowercase English letters, 10 digits, 15 punctuation symbols, and a space, leaving it very few spare bytes for that sort of thing.)

Figuring these out, along with bytes that act as various types of control codes (newline, end message, etc.) is generally a process of trial and error. A good starting point is finding something that's more obviously text, finding where it displays in the game, and comparing the two. Editing the data to see what the game does with it is another way of getting more useful information.

This game's ASCII-based encoding made it possible to find useful text just by skimming through the data in a standard hex editor. For games that use different encodings, specialized editors intended for this sort of thing tend to have a "relative search" function that work on the assumption that 'a', 'b', 'c', and so on have consecutive byte values. Searching for "cafe" with one of these, for example, will find sequences of bytes with values {x+2}{x}{x+5}{x+4}, which is usually enough to find something useful.

(Finding Japanese text with unknown encoding works similarly, but can be trickier. Many games use the typical あいうえお... sequence, but others got more creative and came up with sequences like あアいイうウ... or あぁいぃうぅ... or various other orderings that, okay, sure, do make sense in their own way, but are completely nonstandard. And that's not even getting into kanji. If the game uses any English words or phrases in its text anywhere, which is more common than you might expect, it may be less trouble to start by searching for those.)

In any case, with the single-byte codes figured out, adding those to our table results in something slightly more readable:

Y{$11}3<$01=NEXT>
<$00=END>
1{$04}{$04}7{$0A}fIs {$06}5 {$05}y ok,<br>
{Maxim}? {$05}“ a lot<br>
of money.<$01=NEXT>
aIt {$06}@ {$06}wes us {$06}ta<br>
{$06}N {$0A}ma"y's {$06}s. is,<br>
{$05}{$7F} {$06}mo.<$01=NEXT>

Multi-byte compression


That's starting to look a bit like actual dialog, but there's still a lot missing. This brings us to the second type of compression: two-byte codes. In the Japanese game, these are used to represent kanji (ideograms originally adapted from Chinese). English obviously doesn't have those, so many localized games that include kanji in their Japanese version repurpose the two-byte code system for text compression in the English version.

Notice how the text above has quite a few $05 and $06 bytes? In this game, those are the first bytes of two-byte codes. The English version encoding uses these to encode 512 different three- to fifteen- character words using only two bytes each, from $0500 for "Congratulations" to $06FF for "She". (On a side note, the Japanese version treats $07 similarly to $05 and $06, allowing for up to 255 additional kanji codes, of which 121 are used. However, $07 goes unused in the English version.)

Figuring these codes out is helped by the fact that the game needs a way of knowing what they stand for, too. Its data includes a listing, in plaintext, of all the compressed strings, one after another. (There's also a listing for the one-byte codes that comes right after, but the strings for that are so short that it would be nearly impossible to find it by searching without already knowing its contents in the first place.)

With that determined, we can add those to our encoding table and try again:

Y{$11}3<$01=NEXT>
<$00=END>
1{$04}{$04}7{$0A}fIs that really ok,<br>
{Maxim}? That's a lot<br>
of money.<$01=NEXT>
aIt just tells us how<br>
good {$0A}ma"y's work is,<br>
that's all.<$01=NEXT>

More complicated compression


We're getting close, but there are still two major issues. First, there's all the apparent garbage at the beginning, plus the apparently extraneous 'f' and 'a' that prefix the mostly-readable text. Those are easy enough to edit out when copying, though, so we'll worry about them later.

A more pressing concern is the awkward {$0A}ma" that's still in the middle of one of the lines. There's yet another form of compression going on here! If you guessed that it's a three-byte compression code, you'd be mostly right. This one isn't nearly as straightforward as the others, though.

Three-byte codes starting with $0A form what I've previously called something of a sophisticated ditto mark. Rather than each combination of values referring to a single predetermined string of text, the two bytes after the $0A (without getting any further into the technical details) basically tell the script engine to copy X characters of text from Y bytes earlier in the script.

(The Japanese version, incidentally, applies these $0A codes far more extensively and effectively than the English version, which uses them only infrequently and never copies anything that contains two-byte codes. I have to wonder if there was an automated tool involved that wasn't available, or simply wasn't used, for the localization.)

This isn't something a basic script dumper is capable of handling. As a result, I eventually ended up writing a simple program that would process just the $0A codes, and then ran the output from that through a regular script dumper:

Y{$11}3<$01=NEXT>
<$00=END>
1{$04}{$04}7oo going, Guy!s that really ok,<br>
{Maxim}? That's a lot<br>
of money.<$01=NEXT>
aIt just tells us how<br>
good Jaffy's work is,<br>
that's all.<$01=NEXT>

And now we've finally—wait, what happened to the beginning of the first line of previously-readable text? Although not shown here, I also found that similar garbling occurred in the middle of certain lines, seemingly at random, at least at first glance.

If you look at the more basic dumps, you'll notice that there's an $0A byte in the apparent garbage before the first readable line. Similarly, all the mid-text garbling corresponded to $050A and $060A (and $070A in Japanese) two-byte codes. In these particular cases, the $0A byte is part of a longer code, so does not indicate a lookback and should not be handled as one. But the crude $0A decoder program has no understanding of context and no possible way of knowing that it shouldn't be doing its thing there.

This brings us back to something we've been ignoring, but has become a more important question: What exactly is all that "garbage" that so often appears before and between segments of readable text? It must have some purpose, or it wouldn't be there, right?

Scripting data


Text isn't all of the data that a game needs to create a scene. Characters arrive, leave, and move around; parts of the map may shift and change; animations and sound effects play; the background music may stop or switch tracks. Besides text, the game also needs data that tells it what to do for all of that.

Most of the games I've dealt with so far put their scripting data in one place, and keep their text data sequestered by itself in its own separate area that contains only text and nothing else. When the script needs to display text, it will have an instruction that basically says, "show the text that's stored at offset $6B39", or something along those lines. That's convenient for editing the script and the text semi-independently of each other, and is particularly handy when the text needs to be translated, for example. (And, incidentally, makes things easier on anyone trying to extract the text!)

Lufia II doesn't do it that way. The script itself and the text that it displays are stored together, though they're at least grouped by map and then by NPC or event trigger. All the "junk" that we've been seeing in the dumps so far is actually scripting data! And since most bytes have one meaning as a scripting instruction and a different one as text, this also means that any fully functional parser needs to know when and how to switch between script and text modes.

So, after finding some technical documents by Relnqshd, including one that explains a significant number of the script codes, I finally decided to make an attempt at writing a proper script parser, not just something to dump text. The result for that same block of data now looks more like this:

{$59 11: Fade in from black, speed(?) 17.}
{$33 01 00: Apply movement path #1 to Actor $00 (Maxim).}
{$31 04 04: Actor $04 (Tia) faces right until next movement.}
{$37 0A: Brief delay, lasting 10 frames.}
{$66: Tia speaks:}
    Is that really ok,<br>
    {Maxim}? That's a lot<br>
    of money.<$01=NEXT>
{$61: Maxim speaks:}
    It just tells us how<br>
    good Jaffy's work is,<br>
    that's all.<$01=NEXT>

Now, it's far from perfect. My understanding of the script codes remains limited, so there are still quite a few bytes that the program doesn't know what to do with at all (except possibly to group the arguments with the instructions), along with others that are little more than placeholders. And, of course, bytes taken out of context will still be misinterpreted, like with the crude $0A parser, with every additional value the program tries to parse having the potential both to create new ways for that to happen (by interpreting additional byte values) and to avert old ways (by "consuming" additional instruction arguments before the parser tries to interpret them).

It's still a big improvement, though, and it's been fascinating to discover how some of the scenes work! I'll probably explain at least a few of them in future posts.