跳转至

v0.5.5

Patch release for ParallelODPSManager.pull() safety and no-key parallel pull support.

What Changed

  • Added pull_split_strategy to ParallelODPSConfig:
  • "auto": use hash bucketing when unique_key is provided; otherwise use ROW_NUMBER staging.
  • "hash": force hash bucketing and require unique_key.
  • "row_number": force ROW_NUMBER staging, even if a unique_key is present.
  • Added ROW_NUMBER staging pull for SQL logic that does not expose a stable unique_key. SMF now materializes the base SQL once with {chunk_filter}=1=1, adds an internal row-number column, then pulls chunks by row-number modulo.
  • Added row_number_order_by, row_number_col, and validate_unique_key configuration fields.
  • Added a hard guard that pull() SQL templates must contain {chunk_filter}. Missing placeholders now fail before any ODPS query runs, preventing repeated full-table pulls.
  • Added hash-mode probe validation so invalid or out-of-scope unique_key expressions fail before concurrent chunks start.
  • ROW_NUMBER chunk outputs drop the internal row-number column before writing local CSV.
  • Tightened staging cleanup behavior for success and failure paths, while preserving keep_tmp_on_error=True for debugging.

Migration Notes

  • Existing hash-based calls with a valid unique_key and {chunk_filter} keep the same behavior, except they now run a lightweight validation probe first by default.
  • SQL templates missing {chunk_filter} now raise ValueError. Add AND {chunk_filter} to the intended split scope.
  • Calls without unique_key no longer fail immediately in auto mode; they use ROW_NUMBER staging. This is safer than single-thread fallback but creates a temporary ODPS table, so hash mode is still recommended when a stable key exists.
  • If ROW_NUMBER order needs to be reproducible, pass row_number_order_by="apply_time, flow_id" or another stable expression.

Validation

  • Added pytest coverage for missing {chunk_filter}, hash probe validation, invalid unique_key, ROW_NUMBER staging, chunk-size based staging counts, cleanup policies, and existing push behavior.
  • Full local pytest suite passed before release: 592 passed.
  • Local docs build passed with mkdocs build --strict.