These operator-only environment controls support a controlled comparison of
initial-reset SIGSEGV failures. They do not change tools, prompts, model inputs,
physics, rendering, action budgets, reset arguments or startup timeouts. They
apply to the benchmark RobodojoRoundMixin._reset_environment path, including
round restarts/layout relaunches; they do not change the separate free-play
reset implementation or the 120-second environment-construction diagnostic.
| Environment variable | Default | Values and effect |
|---|---|---|
ORA_ROBODOJO_RESET_TRACEBACK |
on |
on: 60-second stack timer; off: do not arm that timer |
ORA_ROBODOJO_RESET_FATAL_TRACEBACK |
off |
on: install an independent Python fatal-signal traceback file for the duration of reset, if no Python fatal handler already exists |
Only the exact values on and off are accepted; a typo is a configuration
error before reset, not a silent assignment to the control group. The variables
are explicitly allowlisted through the environment RPC launcher. Set them on
the allocated router task/entry process so Supervisor and worker inherit them;
no RunSpec or baseline change is required. No credentials are added to the
worker's environment.
For the timer comparison, keep fatal logging identical in both groups:
# Control: historical timer, plus opt-in fatal log shared by both groups.
export ORA_ROBODOJO_RESET_TRACEBACK=on
export ORA_ROBODOJO_RESET_FATAL_TRACEBACK=on
# Treatment: same setup, only the timer changes.
export ORA_ROBODOJO_RESET_TRACEBACK=off
export ORA_ROBODOJO_RESET_FATAL_TRACEBACK=on
Use the same task/layout, code, concurrency and runtime allocation policy in both groups. Do not mix a changed concurrency cap into the timer experiment. Default production behavior remains timer on, fatal log off. The evaluation owner submits real runs; this feature does not allocate resources or retry them.
Stack timers
stack_timer in ora/environment/backends/robodojo/reset_diagnostics.py writes every
thread's Python stack once, if the block outlasts its time: 60 seconds around a
benchmark reset, 120 around environment construction and free-play resets. It runs on a
Python thread holding the GIL, so a C call that keeps the GIL delays the dump, and it
never ends the run.
These timers used to be faulthandler.dump_traceback_later, which reads other threads'
frames without the GIL. That crashed workers: the reset comparison above was opened for
SIGSEGVs at reset, and a later batch lost 5 of 38 firings the same way, each with
worker_returncode -11 and a dump cut off mid-frame. Plain Python 3.11 reproduces it
when code objects and threads come and go under the dump.
Evidence and limits
Before initial reset, the worker writes
native/reset-initialization-stack.diagnostics.json with schema
ora.robodojo-reset-diagnostics.v1. It records PID, start time, effective timer
setting/60-second duration, fatal request and effective handler, artifact names,
current soft/hard RLIMIT_CORE in bytes (-1 means unlimited), the Linux
core_pattern when readable, and outcome=in_progress. On normal return it
adds finish time and outcome=completed; a Python exception records
outcome=exception and its type, then re-raises the same exception. A hard crash
leaves the pre-reset record; in_progress alone is not proof of a crash—check
environment-session.json.worker_returncode and the run result.
The timer file remains reset-initialization-stack.txt (empty when disabled).
The optional fatal log is reset-initialization-stack.fatal.txt. Round resets
use the corresponding reset-initialization-stack-round-XXXX stems. The
handler is disabled before its file closes; only the handler installed by this
scope is removed. If a Python fatal handler already exists, it is preserved
and evidence says fatal_handler=existing_handler_preserved, fatal_file=null.
An installation error is recorded as setup_failed without preventing reset.
Evidence-write I/O failures log a warning and do not mask reset's own outcome.
This is a Python fatal traceback, not a native C/C++ backtrace or guaranteed
core dump. A crash in a dumper/handler may still truncate it. The code does not
raise process core limits, alter core_pattern, or configure a node collector.
Use the recorded limits and the operator's existing core collection facility
if a native backtrace is needed. No crash report is uploaded by this feature.
The helper is included in both worker and motion source identities. Existing qualification identities may therefore warn of drift under Q32; do not hide that evidence. Default model-visible content is unchanged. CPU tests cover timer defaults/off mode, reset/relaunch arguments, exception identity and file lifetime, preservation of an existing handler, environment forwarding, and a deliberately signaled child process with core dumping disabled. They do not establish whether turning off the timer fixes the real simulator crashes.