Create initial logic for seperating maz table from taz table by lachlan-git · Pull Request #215 · BayAreaMetro/tm2py

lachlan-git · 2025-10-20T21:49:35Z

What existing problem does the pull request solve and why should we include it?

PR created to address issue #210 and #212

Separating / re-implementing MAZ travel demands and integrating with assignment in Emme without storing an intermediate OMX file.

@lachlan-git, please update the PR description to include brief descriptions for each work item, use checkbox to track progress. For example:

What is the testing plan?

Rerun household and highway_maz_assign, the following conditions should be met:

Total Maz Demand originating at MAZ node should match household demand
All maz trips should be less than XX miles
For Taz Total Centroid Connector Flows should equal twice the total demand for each class
For Taz, we should see very similar demands (less the maz trips) with before changes are implemented.

Code formatting

will be PEP8 Compliant on all changed files

Applicable Issues

Please do not create a Pull Request without creating an issue first.

Put closes #XXXX in your comment to auto-close the issue that your PR fixes.

closes 🐛 Bug: 0 MAZ -MAZ demand #210
closes 🐛 Bug: Setting MAZ Auto Flow as Background Traffic #212

lachlan-git · 2025-10-20T21:59:09Z

Notice that in the original implementation is meant to treat ride shares differently, but this code path is never touched:

tm2py/tm2py/components/demand/prepare_demand.py

Line 545 in 50e9009

elif trip_mode in [15, 16]:

Does this need to be changed? Should 15, 16 be removed from the list above

lachlan-git · 2025-10-20T22:02:36Z

The threshold for whether the trip from CTRAMP becomes a maz trip or a taz trip is in the "highway" section of the model config

    # threshold (miles) where a drive trip is short enough to be moved to a taz
    maz_drive_distance_threshold = 0.5

This will require a new file in tm2py-utils.

tm2py/components/demand/household.py

i-am-sijia · 2025-10-21T15:59:35Z

tm2py/components/demand/prepare_demand.py

-            it = it_full[it_full.time_period == time_period]
-            jt = jt_full[jt_full.time_period == time_period]
+            it_both_taz_and_maz = it_full[it_full.time_period == time_period]
+            jt_both_taz_and_maz = jt_full[jt_full.time_period == time_period]


I understand that whoever first authored this function did not do this - can we please be more intentional with object naming? E.g., if it is a dataframe, put a _df suffix in the name please.

will keep naming convention for the code I write, I wont refactor the entire function

Ok, please do that.

tm2py/components/demand/prepare_demand.py

i-am-sijia · 2025-10-21T16:32:04Z

Notice that in the original implementation is meant to treat ride shares differently, but this code path is never touched:

tm2py/tm2py/components/demand/prepare_demand.py

Line 545 in 50e9009

elif trip_mode in [15, 16]:

Does this need to be changed? Should 15, 16 be removed from the list above

Good catch. It does seem problematic. If 15-taxi and 16-tnc are removed from the list above, do they not get created in the output trip tables? I think that is fine if the intention here is to reclassify taxi and tnc and assign them as part of the DA, HOV2, HOV3 trip tables.

tm2py/canonical_mode_choice.py

lachlan-git · 2025-11-04T20:45:16Z

Notice that in the original implementation is meant to treat ride shares differently, but this code path is never touched:

tm2py/tm2py/components/demand/prepare_demand.py

Line 545 in 50e9009

elif trip_mode in [15, 16]:

Does this need to be changed? Should 15, 16 be removed from the list above

Good catch. It does seem problematic. If 15-taxi and 16-tnc are removed from the list above, do they not get created in the output trip tables? I think that is fine if the intention here is to reclassify taxi and tnc and assign them as part of the DA, HOV2, HOV3 trip tables.

Code seemed stale, has some updates to be reviewed

i-am-sijia

I know this is still a WIP. Just leaving some incremental comments before you finalize the code changes. We should also run the model through to confirm code changes do not break things.

i-am-sijia · 2025-11-07T22:12:33Z

tm2py/config.py

    interchange_nodes_file: str = Field()
    apply_msa_demand: bool = True
    reliability: bool = Field(default=True)
+    maz_drive_distance_threshold: float = Field()


If we are adding a new paramemter to the model config, we should create a corresponding PR with actual config changes to tm2py-utils.

i-am-sijia · 2025-11-07T22:14:51Z

tm2py/config.py

    taxi_split: Dict[str, float]
    single_tnc_split: Dict[str, float]
    shared_tnc_split: Dict[str, float]
-    ctramp_mode_names: Dict[float, str]


Same here, make sure this is in a PR to tm2py-utls. We should also run the model after deleting this from the model config to make sure it does not break anything.

i-am-sijia · 2025-11-07T22:16:06Z

tm2py/components/network/highway/highway_assign.py

    def _copy_maz_flow(self):
        """Copy maz_flow from MAZ demand assignment to ul1 for background traffic."""
-        self._network_calculator("ul1", "0")
+        self._network_calculator("ul1", "@maz_flow")


Have this been tested to confirm there is background flow in the network?

The testing of the background maz flows. was done by comparing the input demands (intermediatly saved as a csv) and the total volumes (inflow or and outflow) of the MAZ connectors on the network. It was noticed that more flow was missing than expected.

Note: On review of the code, this is likely because the code filters out MAZ trips longer than 10 miles.
https://github.com/BayAreaMetro/tm2py/blob/d454ee0b7aae02b8d454431ccc11077c2f8c2a5a/tm2py/components/network/highway/highway_maz.py#L52C1-L52C20

2 changes improved this, but did not completely mitigate this discrepency.

by using a TAZ matrix (instead of ct-ramps internal distance) implemented in fe20e44. There were no trips over the 10 miles this lead. to incomplete demand assignment

self trips (starting and ending at the same maz node) were also filtered out of the maz_demands, as these do not appear on the network.

i-am-sijia · 2025-11-07T22:21:53Z

tm2py/components/network/highway/highway_maz.py

                        )
                        continue
-                    self._process_demand(time, i, maz_ids)
+                    self._process_demand(time, i, maz_ids, model_id_to_maz)


I'm not a fan of calling it model_id_to_maz. Let's be as explicit as we can and avoid ambiguity. There are three types of IDs: MAZ labels (county-based), MAZ sequentials (starts from 1 and used by CT-RAMP), MAZ EMME node IDs. Maybe we should call them labels, sequentials, and emmes. Please coordinate with @yueshuaing to confirm the final namings and mappings.

Unfortunately, reviewing the tests this week, it appears there may still be some crosswalk issues,

tm2py/components/demand/prepare_demand.py

i-am-sijia · 2025-11-07T22:29:38Z

tm2py/components/demand/prepare_demand.py

-            it = it_full[it_full.time_period == time_period]
-            jt = jt_full[jt_full.time_period == time_period]
+            individual_trip_both_taz_and_maz_df = it_full[it_full.time_period == time_period]
+            joint_trip_both_taz_and_maz = jt_full[jt_full.time_period == time_period]


joint_trip_both_taz_and_maz_df?

tm2py/components/demand/prepare_demand.py

lachlan-git · 2025-11-10T19:59:36Z

@i-am-sijia The person trips get converted into vehicle trips at this point in the code

tm2py/tm2py/components/demand/prepare_demand.py

Line 306 in 237689d

# 2) Convert Person Trips into Vehicle trips

i-am-sijia · 2025-11-10T21:05:13Z

@i-am-sijia The person trips get converted into vehicle trips at this point in the code

Thank you. CTRAMP used 0.5 to convert SR2 person trips into vehicles, and 0.3 for SR3. [1][2]. I do not like how this is hardcoded in tm2py. Shall we expose it to the config?

Ennazus · 2025-11-13T16:21:26Z

Oh wow! You've been doing lots of work here which is great. Could you write me up a couple paragraphs to describe what you did and what you're still working on?

not being exported properly

i-am-sijia · 2025-11-14T21:06:38Z

bug

i-am-sijia · 2025-11-18T03:57:32Z

Update (11/17). Uncompleted items:

MAZ assignment reports that there are OD pairs that are more than 0.5 miles apart (e.g., 10 miles), which should not happen because they should have been filtered out in:

tm2py/tm2py/components/demand/prepare_demand.py

Lines 522 to 536 in 659ad08

    
           it_is_maz_trip = ( 
        
               (individual_trip_both_taz_and_maz_df["trip_dist"] < self.controller.config.highway.maz_drive_distance_threshold) & 
        
               individual_trip_both_taz_and_maz_df["trip_mode"].isin(DRIVE_MODES) 
        
           ) 
        
           it_maz_df = individual_trip_both_taz_and_maz_df[it_is_maz_trip] 
        
           it = individual_trip_both_taz_and_maz_df[~it_is_maz_trip] 
        
           jt_is_maz_trip = ( 
        
               (joint_trip_both_taz_and_maz_df["trip_dist"] < self.controller.config.highway.maz_drive_distance_threshold) & 
        
               joint_trip_both_taz_and_maz_df["trip_mode"].isin(DRIVE_MODES) 
        
           ) 
        
           jt_maz_df = joint_trip_both_taz_and_maz_df[jt_is_maz_trip] 
        
           jt = joint_trip_both_taz_and_maz_df[~jt_is_maz_trip] 
        
           maz_trips = pd.concat([it_maz_df, jt_maz_df])

, based on the trip_dist column in the trip list from ct-ramp. This suggests either the sequential MAZ ID to EMME node ID correspondence is wrong (unlikely because it's recently updated in PR Add MAZ data model #216), or the trip_dist value in the trip list is wrong. To check the latter, we know that ct-ramp appends TAZ skims to the trip list https://github.com/BayAreaMetro/travel-model-two/blob/e941242fc43fce60beea24be6263f1cd1aae231b/src/java/com/pb/mtctm2/abm/reports/SkimBuilder.java#L234-L310, we therefore appended TAZ drive alone skims to the trip list to verify the trip_dist column. The snapshot below confirms that there are some large discrepancies between the trip_dist column and the appended TAZ drive alone skims for some MAZ pairs. We need to investigate why this is happening. Could it be because of the HOV and/or toll distance skims?

MAZ auto trip demand for MAZ assignment for Early Morning. Note the difference in trip_distance and taz_distance ("EA_DA_dist")

This may be related to item No.1. When comparing the MAZ demand against the MAZ assigned volume, we found that the assigned volume is always less than the demand. We need to investigate why this is happening.
Create initial logic for seperating maz table from taz table #215 (comment)
We would like to get rid of writing out TAZ vehicle OD matrices to disk as OMX files, and instead write them directly to Emmebank. This would require modifying the demand preparation step to write directly to Emmebank instead of writing to OMX first, for resident demand, auxiliary demand models including airport, truck, IE/EI, etc., and then modifying the highway assignment step to read from Emmebank instead of OMX.

maz demands

lachlan-git added 2 commits October 20, 2025 15:13

initial commit

cab739d

maz table is manually separated from taz

055efea

lachlan-git requested a review from i-am-sijia October 20, 2025 21:49

lachlan-git self-assigned this Oct 20, 2025

i-am-sijia reviewed Oct 21, 2025

View reviewed changes

tm2py/components/demand/household.py Outdated Show resolved Hide resolved

i-am-sijia reviewed Oct 21, 2025

View reviewed changes

tm2py/components/demand/prepare_demand.py Outdated Show resolved Hide resolved

i-am-sijia reviewed Oct 21, 2025

View reviewed changes

tm2py/components/demand/prepare_demand.py Outdated Show resolved Hide resolved

lachlan-git added 3 commits October 22, 2025 13:58

I believe new demands method is working

506a8fb

This version works with multiple iterations

b29c412

working out distance

20b5b02

This comment was marked as duplicate.

Sign in to view

lachlan-git added 6 commits November 3, 2025 18:00

assignment is working

61bbbf8

add canonical mode choice file

34a2079

fixed up some naming according to comment

cffca86

corrected set background maz traffic

b11e24e

fixed typing highlighted by vscode

32f5ee1

fixing minor errors made

cd7b704

i-am-sijia reviewed Nov 4, 2025

View reviewed changes

tm2py/canonical_mode_choice.py Outdated Show resolved Hide resolved

finalized logic for mode demand preporation

2d87648

lachlan-git added 5 commits November 4, 2025 15:46

was missing school bus in previous enum

ed32856

removed commented line

8570808

was missing some demands

beb77b8

removed debug statements

038e431

one more debug statement

7619c9c

i-am-sijia reviewed Nov 7, 2025

View reviewed changes

lachlan-git added 2 commits November 10, 2025 12:14

fixed incorrect enum translation

b7aeb09

added requested comments

237689d

made more permanent location for maz demands

9d1870a

lachlan-git and others added 3 commits November 13, 2025 16:55

fixed bug in deferred write leading to taz demands

ebe04e0

not being exported properly

Add explicit telecommute documentation

1a9cfb6

Merge branch 'develop' into update_maz_seperation_logic

fdcfae2

merged single source of truth maz, fixed maz ass

d454ee0

bug

lachlan-git mentioned this pull request Nov 17, 2025

Update maz assignment BayAreaMetro/tm2py-utils#16

Merged

review comments

659ad08

added reference to taz skims when outputting

fe20e44

maz demands

i-am-sijia mentioned this pull request Nov 20, 2025

🧹 Chore: Remove County Groups in MAZ Assignment #218

Open

3 tasks

Conversation

lachlan-git commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What existing problem does the pull request solve and why should we include it?

What is the testing plan?

Code formatting

Applicable Issues

Uh oh!

lachlan-git commented Oct 20, 2025

Uh oh!

lachlan-git commented Oct 20, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

i-am-sijia commented Oct 21, 2025

Uh oh!

This comment was marked as duplicate.

Uh oh!

lachlan-git commented Nov 4, 2025

Uh oh!

i-am-sijia left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lachlan-git Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lachlan-git commented Nov 10, 2025

Uh oh!

i-am-sijia commented Nov 10, 2025

Uh oh!

Ennazus commented Nov 13, 2025

Uh oh!

i-am-sijia commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

i-am-sijia commented Nov 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lachlan-git commented Oct 20, 2025 •

edited

Loading

lachlan-git Nov 17, 2025 •

edited

Loading

i-am-sijia commented Nov 14, 2025 •

edited

Loading