https://github.com/uber/pyflame + https://github.com/brendangregg/FlameGraph shows that for some crawls, 12-30% of the time is spent in `add_cookie_header`, which does a lot of URL parsing and clears the cookie jar too often (of which one component is a slow `deepcopy`). Note: the configure step for `pyflame` is ``` PKG_CONFIG_PATH=~/.pyenv/versions/3.7.0/lib/pkgconfig ./configure ``` with https://github.com/uber/pyflame/pull/153 applied [flamegraphs.zip](https://github.com/ludios/grab-site/files/2404769/flamegraphs.zip)
https://github.com/uber/pyflame + https://github.com/brendangregg/FlameGraph shows that for some crawls, 12-30% of the time is spent in
add_cookie_header, which does a lot of URL parsing and clears the cookie jar too often (of which one component is a slowdeepcopy).Note: the configure step for
pyflameiswith uber-archive/pyflame#153 applied
flamegraphs.zip