Starting preliminary 0.94 testing - "Headless fullnode"

[Roy Badami]

Could well be just chance since the behaviour before was clearly somewhat nondeterministric.  But I did the above and it synced my wallets perfectly first time (and I hadn't managed to get a successfully sync at all so far up till now).  If I get the chance tomorrow I'll nuke the databases and try again to confirm whether this is repeatable.

Well yes and no. I expect the issue is object life span: one type of threads creates some data that gets cleaned up before the next type of threads is done with it. If you reduce the amount of threads per groups to the bare minimum, you also reduce the "surface area" for the bug and thus the chance it will occur.

Just to confirm, only change I made was as you suggested:

That's the proper change to force thread count for all groups down to 1.

Is there any point in my trying the new version or will this version just behave the same for me?

I'd like you to run it, the main fix in this version is supposed to cover your issue.

Ah, glad I asked, then :-)

I initially assumed that since I wasn't seeing a segfault there wasn't much point in trying it.  I'll give it a go and report back.

[1715155384]

1715155384

[1715155384]

1715155384

[1715155384]

1715155384

[Carlton Banks]

I heard there were new processor instructions to abstract a bit of the complexity away from the programming logic, but Intel screwed up the implementation somehow and it's not quite "on the streets" yet. But all the specs are finalised as far as I remember, I believe they're calling it "lock elision" in C++, although I may be remembering wrong. So there's some more C++ to render the old standard obsolete for us all to look forward to  .

That's pretty fancy. It doesn't really simplify much. It looks like it's meant to speed up interlocked operations that don't overlap in memory, and it's only in some Haswell CPUs so far. I suspect it's the kind of extra assembly optimization that I will personally never have to use, as either compilers or OSes will upgrade their locks with HLE. The way it looks like, there is little reason not to standardize it for every lock operation.

It's supposed to be some kind of speculative execution; the majority of the time that a lock is applied, there are no writes to data structures that break the intention of the code. I forget the full details though, this was years ago. Haswell was where it was borked, it was disabled in the firmware for all lines IIRC (TSX is what Intel call it). Suposedly the next 16nm Intel chip out in Q4 this year has it working correctly.

Just to totally clarify, I cannot get this or the build from your last commit to behave in any way except for this bug. It does this every time I load it up. Carefully deleting old Db and logs every run. Tested that Armory 93.1 & 93.2 work normally on the same VM (I cloned it from a working bitcoin-configured VM to begin with, but you never know).

And that bit is what's disturbing me considering this part uses the same code across version. Again, this is pretty old code, so I'm puzzled as to why it starts failing now. Anyways, I think I got a clue as to what's going on.

Like I said, this didn't happen in the first couple of 0.94 pre-versions you came up with, not sure what you can infer from that.

[goatpig]

Like I said, this didn't happen in the first couple of 0.94 pre-versions you came up with, not sure what you can infer from that.

The usual conclusion in this case is: there always was an issue with this code, and changes in its vicinity revealed it. Otherwise it was probably occurring at low frequency, passing itself as a scan segfault of sorts.

[Roy Badami]

Like I said, this didn't happen in the first couple of 0.94 pre-versions you came up with, not sure what you can infer from that.

The usual conclusion in this case is: there always was an issue with this code, and changes in its vicinity revealed it. Otherwise it was probably occurring at low frequency, passing itself as a scan segfault of sorts.

The usual case is "OMG how can that ever have worked?" :-)

[Roy Badami]

Ok, first attempt got a considerable way through scanning transactions, then failed with a popup "BDM not ready"

This was in the logs

Code:

-INFO  - 1434747557: (BlockUtils.cpp:1443) checking scan integrity
-WARN  - 1434747557: (BlockUtils.cpp:1448) Top scanned block does not match top block header
-ERROR - 1434747557: (BlockUtils.cpp:1464) shouldnt get to this line
-INFO  - 1434747557: (BlockDataViewer.cpp:155) Enabling zero-conf tracking 
-ERROR - 1434747557: (BDM_mainthread.cpp:429) BDM thread failed: BDM is not ready!

EDIT: Before we got to this point we had two instances of:

Code:

-WARN  - 1434747337: (BlockUtils.cpp:1448) Top scanned block does not match top block header
-WARN  - 1434747337: (BlockUtils.cpp:1499) Issue is significant, attempting repairs

[goatpig]

Thanks, that's enough information for now

[Roy Badami]

Ok, second attempt and it ran all the way through successfully.  Had eleven instances of the 'top scanned block' issue in the logs, but got all the way through and displays the correct balances.  

EDIT: Both runs were with no existing databases directory.

[goatpig]

Had eleven instances of the 'top scanned block' issue in the logs

This is a marker that the multi threaded block pulling code is still somehow messy, i.e. the scan fails to pull data for a block and halts. The fixing part detects this, goes over all blocks past that point, realizes there is nothing wrong with the data (or there is and it tries to fix it) and resumes scanning from the highest committed block, on and on.

Also the "attempting fix" code is something I have to go over, it's the stuff from 0.93 that is not really on point with the 0.94 changes.

[Roy Badami]

Also the "attempting fix" code is something I have to go over, it's the stuff from 0.93 that is not really on point with the 0.94 changes.

Ah.  Hence the 'should not get here' error?

[goatpig]

Also the "attempting fix" code is something I have to go over, it's the stuff from 0.93 that is not really on point with the 0.94 changes.

Ah.  Hence the 'should not get here' error?

That one is somewhat nastier. It triggers if the the DB cannot find the hash of the top scanned block, which probably means the scan halted before it failed to commit any blocks at all (and returned an empty top scanned block hash). It could be the concurrency issue triggering early on or something unrelated, although I expect it's the former.

[goatpig]

Here's a tentative fix for Roy's issue. I added some minor changes around the part Carlton is stuck on but I don't expect this is it yet. Pull away.

[Carlton Banks]

You have gain a little traction. Latest:

Code:

-INFO  - 1434792421: (BlockUtils.cpp:345) parsing headers in file 284
-INFO  - 1434792421: (BlockUtils.cpp:345) parsing headers in file 285
-DEBUG - 1434792422: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1434792424: (BlockUtils.cpp:1385) Looking for first unrecognized block
-INFO  - 1434792424: (BlockUtils.cpp:1389) Updating Headers in DB
-INFO  - 1434792438: (BlockUtils.cpp:1600) Loading block data... file 0 offset 0
-INFO  - 1434792438: (BlockUtils.cpp:395) reading blocks from file 0
-INFO  - 1434792441: (BlockUtils.cpp:395) reading blocks from file 1

*snip snip*

-INFO  - 1434792231: (BlockUtils.cpp:395) reading blocks from file 285
-INFO  - 1434792232: (BlockUtils.cpp:1403) Wrote blocks to DB in 83.7816s
-INFO  - 1434792232: (BlockUtils.cpp:1443) checking scan integrity
-WARN  - 1434792232: (BlockUtils.cpp:1448) Top scanned block does not match top block header
-ERROR - 1434792232: (BlockUtils.cpp:1464) shouldnt get to this line
-INFO  - 1434792232: (BlockDataViewer.cpp:155) Enabling zero-conf tracking 
-ERROR - 1434792232: (BDM_mainthread.cpp:429) BDM thread failed: BDM is not ready!

Still failing, but getting further through. "BDM not ready" appears as a warning dialog box, OK'ing that shuts Armory. Still a deterministic fault, so far, so part of the same issue maybe?

[goatpig]

Still failing, but getting further through. "BDM not ready" appears as a warning dialog box, OK'ing that shuts Armory. Still a deterministic fault, so far, so part of the same issue maybe?

This means you are through your original issue. I would like you to run it a few times to determine whether you are completely past that point or this was a fluke.

As for the error, you are on the same thing as Roy, although the nastier form. From the looks of it, it didn't even try to scan anything! Do you have wallets registered?

[Carlton Banks]

Still failing, but getting further through. "BDM not ready" appears as a warning dialog box, OK'ing that shuts Armory. Still a deterministic fault, so far, so part of the same issue maybe?

This means you are through your original issue. I would like you to run it a few times to determine whether you are completely past that point or this was a fluke.

As for the error, you are on the same thing as Roy, although the nastier form. From the looks of it, it didn't even try to scan anything! Do you have wallets registered?

No wallets in that run, been trying with and without. With wallets, logging output ends with the same error, and it also produces a little stats for the wallets threads that you were expecting to see there. I'll try again to confirm this, only did a few tries of each earlier on.

[Carlton Banks]

Previous bug seems to be squashed. Several trials and no sign of it.

BDM not ready error isn't consistent for me, I just added wallets to the armory home folder and it almost completed correctly... the scan finished, and the application was functioning fine. What didn't work was the shutdown, I got the same response as with the previous bug. Then when I tried re-opening Armory with the built database:

Code:

********************************************************************************
Loading Armory Engine:
   Armory Version:       0.93.99.1
   Armory Build:         470e2001e8
   PyBtcWallet  Version: 1.35
Detected Operating system: Linux
   OS Variant            : ('debian', '8.1', '')
   User home-directory   : /home/user
   Satoshi BTC directory : /home/user/.bitcoin
   Armory home dir       : /home/user/.armory
   ArmoryDB directory     : /home/user/.armory/databases
   Armory settings file  : /home/user/.armory/ArmorySettings.txt
   Armory log file       : /home/user/.armory/armorylog.txt
   Do wallet checking    : True
-INFO  - 1434819043: (BlockUtils.cpp:873) blkfile dir: /home/user/.bitcoin/blocks
-INFO  - 1434819043: (BlockUtils.cpp:874) lmdb dir: /home/user/.armory/databases
-INFO  - 1434819043: (lmdb_wrapper.cpp:439) Opening databases...
-INFO  - 1434819043: (BlockUtils.cpp:1228) Executing: doInitialSyncOnLoad
-INFO  - 1434819043: (BlockUtils.cpp:1300) Total number of blk*.dat files: 286
-INFO  - 1434819043: (BlockUtils.cpp:1301) Total blockchain bytes: 38,315,598,682
-INFO  - 1434819043: (BlockUtils.cpp:1738) Reading headers from db
-INFO  - 1434819050: (BlockUtils.cpp:1764) Found 361797 headers in db
-DEBUG - 1434819050: (Blockchain.cpp:214) Organizing chain w/ rebuild
-WARN  - 1434819053: (BlockUtils.cpp:1329) --- Fetching SSH summaries for 491 registered addresses
-INFO  - 1434819054: (BlockUtils.cpp:1342) Left off at file 285, offset 115218060
-INFO  - 1434819054: (BlockUtils.cpp:1345) Reading headers and building chain...
-INFO  - 1434819054: (BlockUtils.cpp:1346) Starting at block file 285 offset 115218060
-INFO  - 1434819054: (BlockUtils.cpp:1348) Block height 361784
-INFO  - 1434819054: (BlockUtils.cpp:345) parsing headers in file 285
-DEBUG - 1434819054: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1434819056: (BlockUtils.cpp:1385) Looking for first unrecognized block
-INFO  - 1434819056: (BlockUtils.cpp:1389) Updating Headers in DB
-INFO  - 1434819056: (BlockUtils.cpp:1600) Loading block data... file 285 offset 115218060
-INFO  - 1434819056: (BlockUtils.cpp:395) reading blocks from file 285
-INFO  - 1434819057: (BlockUtils.cpp:1403) Wrote blocks to DB in 0.091881s
-WARN  - 1434819057: (BlockUtils.cpp:1118) Scanning from 361780 to 361788

(python:14638): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion 'value >= min && value <= max' failed
-INFO  - 1434819057: (BlockUtils.cpp:1443) checking scan integrity
-INFO  - 1434819057: (BlockUtils.cpp:1562) --- bwbDtor: 0s
-INFO  - 1434819057: (BlockUtils.cpp:1563) Scanned Block range in 0.801479s
-INFO  - 1434819057: (BlockUtils.cpp:1566) Finished loading at file 285, offset 117026174
-INFO  - 1434819057: (BlockDataViewer.cpp:155) Enabling zero-conf tracking 

Tried re-opening again, same Gtk-CRITICAL, then BDM not ready was thrown. New blocks didn't register in Armory, not in the logs or the GUI.

[goatpig]

I'm told the Gtk-Critical thing is an issue with the Qt version that ships in wheezy (4.8.2, dates from 2012 or something). I use Debian wheezy for my Linux tests and I see this error all the time, without any issue, so I don't believe it is related to BDM snafus.

I've just pushed a small change to squash a dead lock and get rid of BDM not ready false positive errors. This should make it easier to identify the current state of the scanning bug.

[Carlton Banks]

I'm told the Gtk-Critical thing is an issue with the Qt version that ships in wheezy (4.8.2, dates from 2012 or something). I use Debian wheezy for my Linux tests and I see this error all the time, without any issue, so I don't believe it is related to BDM snafus.

Up to date jessie here, Debian haven't updated much, it seems.

Output from build 3fad5ecd54:

Code:

-INFO  - 1434822571: (BlockUtils.cpp:395) reading blocks from file 285
-INFO  - 1434822572: (BlockUtils.cpp:1403) Wrote blocks to DB in 78.396s
-WARN  - 1434822572: (lmdb_wrapper.cpp:659) Clearing history
-WARN  - 1434822572: (BlockUtils.cpp:1118) Scanning from 0 to 361794
-WARN  - 1434822578: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 2500
-WARN  - 1434822581: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 85000
-WARN  - 1434822584: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 105000
-WARN  - 1434822587: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 115000
-WARN  - 1434822590: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 120000
-WARN  - 1434822592: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 125000
-WARN  - 1434822595: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 127500
-WARN  - 1434822600: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 130000
-WARN  - 1434822603: (BlockWriteBatcher.cpp:2200) Readjusting thread count: 
-WARN  - 1434822603: (BlockWriteBatcher.cpp:2201) 3 readers
-WARN  - 1434822603: (BlockWriteBatcher.cpp:2202) 1 workers
-WARN  - 1434822603: (BlockWriteBatcher.cpp:2203) 1 writers
-WARN  - 1434822603: (BlockWriteBatcher.cpp:2204) 1 old reader count
-WARN  - 1434822603: (BlockWriteBatcher.cpp:2205) 2 old worker count
-WARN  - 1434822603: (BlockWriteBatcher.cpp:2206) 1 old writer count
-WARN  - 1434822605: (BlockWriteBatcher.cpp:494) Finished applying blocks up to 132500
-WARN  - 1434822606: (BlockWriteBatcher.cpp:544) --- feedSleep: 0 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:547) --- workers: 0 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:550) --- writeStxo: 0.001505 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:552) --- writeStxo_grabMutex: 0.000182 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:555) --- waitingOnSerThread: 0.045835 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:557) --- waitForDataToWrite: 51.6417 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:561) --- checkForCollisions: 0.001846 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:563) --- getExistingKeys: 0.002083 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:565) --- getNewKeys: 0.003505 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:567) --- getSSHHeadersLock: 0.00015 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:569) --- computeDBKeys: 0.006044 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:571) --- getSshHeaders: 0.000427 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:573) --- finalizeSerialization: 0.152087 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:578) --- serializeBatch: 0.03542 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:580) --- updateSSH: 0.026719 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:582) --- serializeSubSsh: 0.000365 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:584) --- waitOnSSHser: 0.100496 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:588) --- waitOnWriteThread: 0.000114 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:591) --- putSSH: 0.085506 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:593) --- putSTX: 0.007947 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:596) --- getnextfeed: 36.719 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:598) --- inControlThread: 51.798 s
-INFO  - 1434822606: (BlockUtils.cpp:1443) checking scan integrity
-WARN  - 1434822606: (BlockUtils.cpp:1448) Top scanned block does not match top block header
-WARN  - 1434822606: (BlockUtils.cpp:1513) Issue is significant, attempting repairs
-INFO  - 1434822606: (BlockUtils.cpp:1525) Checking dupIDs from 133240 onward
-WARN  - 1434822606: (BlockUtils.cpp:1118) Scanning from 133240 to 361794
-WARN  - 1434822606: (BlockWriteBatcher.cpp:544) --- feedSleep: 0 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:547) --- workers: 0 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:550) --- writeStxo: 0.001505 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:552) --- writeStxo_grabMutex: 0.000182 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:555) --- waitingOnSerThread: 0.045835 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:557) --- waitForDataToWrite: 51.6417 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:561) --- checkForCollisions: 0.00185 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:563) --- getExistingKeys: 0.004444 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:565) --- getNewKeys: 0.00351 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:567) --- getSSHHeadersLock: 0.00015 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:569) --- computeDBKeys: 0.006044 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:571) --- getSshHeaders: 0.000427 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:573) --- finalizeSerialization: 0.152087 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:578) --- serializeBatch: 0.03542 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:580) --- updateSSH: 0.026719 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:582) --- serializeSubSsh: 0.000365 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:584) --- waitOnSSHser: 0.100496 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:588) --- waitOnWriteThread: 0.000114 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:591) --- putSSH: 0.085506 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:593) --- putSTX: 0.007947 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:596) --- getnextfeed: 36.719 s
-WARN  - 1434822606: (BlockWriteBatcher.cpp:598) --- inControlThread: 51.7981 s
-INFO  - 1434822606: (BlockUtils.cpp:1443) checking scan integrity
-WARN  - 1434822606: (BlockUtils.cpp:1448) Top scanned block does not match top block header
-ERROR - 1434822606: (BlockUtils.cpp:1478) shouldnt get to this line
-INFO  - 1434822606: (BlockDataViewer.cpp:155) Enabling zero-conf tracking 
-ERROR - 1434822606: (BDM_mainthread.cpp:429) BDM thread failed: BDM is not ready!

[goatpig]

Give it a couple more spins see if it always fails at the same block, please.

There is a backport version of Qt 4.8.6, that should get rid of the Gtk issue I think.

EDIT: actually give it a minute before running this again, pushing some extra verbose.

EDIT2: should be commit 0008c05

[Carlton Banks]

From 0008c05:

Code:

-INFO  - 1434835078: (BlockUtils.cpp:1403) Wrote blocks to DB in 78.2405s
-WARN  - 1434835078: (lmdb_wrapper.cpp:659) Clearing history
-WARN  - 1434835078: (BlockUtils.cpp:1118) Scanning from 0 to 361813
-INFO  - 1434835078: (BlockWriteBatcher.cpp:1675) height 0 exceeds scan range, ending scan
-WARN  - 1434835078: (BlockWriteBatcher.cpp:544) --- feedSleep: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:547) --- workers: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:550) --- writeStxo: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:552) --- writeStxo_grabMutex: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:555) --- waitingOnSerThread: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:557) --- waitForDataToWrite: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:561) --- checkForCollisions: 0.001311 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:563) --- getExistingKeys: 0.001773 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:565) --- getNewKeys: 0.00232 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:567) --- getSSHHeadersLock: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:569) --- computeDBKeys: 0.004158 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:571) --- getSshHeaders: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:573) --- finalizeSerialization: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:578) --- serializeBatch: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:580) --- updateSSH: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:582) --- serializeSubSsh: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:584) --- waitOnSSHser: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:588) --- waitOnWriteThread: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:591) --- putSSH: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:593) --- putSTX: 0 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:596) --- getnextfeed: 6e-06 s
-WARN  - 1434835078: (BlockWriteBatcher.cpp:598) --- inControlThread: 0.000199 s
-INFO  - 1434835078: (BlockUtils.cpp:1443) checking scan integrity
-WARN  - 1434835078: (BlockUtils.cpp:1448) Top scanned block does not match top block header
-ERROR - 1434835078: (BlockUtils.cpp:1478) shouldnt get to this line
-INFO  - 1434835078: (BlockDataViewer.cpp:155) Enabling zero-conf tracking 
-ERROR - 1434835078: (BDM_mainthread.cpp:429) BDM thread failed: BDM is not ready!

[Carlton Banks]

It seems to me like the thread count readjustment triggers the BDM not ready's. If it dodges the bug, it adjusts about 3 times during the first 160,000 blocks, then the threads totally smother the CPU cores. Sometimes it doesn't even handle the zeroth adjust, such as the run that generated the output in the previous post.