input track data

[wendellwt]

What is the best way to input lots of track data into PyMEOS?

I have a pandas dataframe with columns of id, lat, lon, and timestamp. The lat and lon are both numpy.float64 and the timestamp is a 'pandas._libs.tslibs.timestamps.Timestamp'.

The following code works, but takes about 8 minutes to create TGeomPointInst for 3,600,000 points (but only 3 seconds to make TGeomPointSeq elements of the 26,000 resulting tracks).

Would making the conversion function into a vectorized numpy function help?

Thank you,
Wendell

from postgis import Point
from pymeos import TGeomPointInst, TGeomPointSeq

def make_tgp(lat, lon, tstamp):
    return TGeomPointInst(point=Point(lon, lat), timestamp=tstamp)

df['tgp'] = df.apply( lambda row: \
           make_tgp(row['lat'], row['lon'], row['posit_time']), axis=1)

grouped_df = df \
    .groupby('flight_index',as_index=False ) \
        .agg( {'fid' : 'first',
               'tgp' : list })

grouped_df['tgps'] = grouped_df['tgp'].apply( lambda t: TGeomPointSeq(instant_list=t))

[wendellwt]

This very nice article had this to say about .apply:

The pandas apply method is very simple and intuitive. However, it is also one
of the slowest ways of applying calculations on large datasets.

We'll try out some of the methods described in that article.

1) convert lat, lon to Point

As a test case, we'll convert the columns of lat, lon to Point using .apply:

df['pt'] = df.apply( lambda row: Point(row['lon'], row['lat']), axis=1)

elapsed time: 4m 30s

Of the methods described in the above reference, we'll use the last one (np.vectorize), which was the fastest:

df['pt'] = np.vectorize(Point)(df['lon'].to_numpy(), df['lat'].to_numpy())

elapsed time: 0m 8s 38ms

Note that the input columns to Point are floats, not objects. I don't know if this allows numpy to achieve such a dramatic improvement.

2) timestamp

The timestamp was converted from tslibs.Timestamp to datetime. Even when using .apply this only took 3 seconds, so was not considered for vectorization.

3) convert all Points and datetimes to TGeogPointInst

Using .apply, this method takes:

df['tgp_a'] = df.apply( lambda row: TGeomPointInst(point=row['pt'], timestamp=row['tm']), axis=1)

elapsed time: 9m 32s

Of the methods described, list comprehension with zip was slightly faster than itertuples:

df['tgp_c'] = [TGeomPointInst(point=pt, timestamp=tm) for pt, tm in zip(df['pt'],df['tm'])]

elapsed time: 3m 52s 11ms

Numpy vectorization was about the same as .itertuples. Was this because the inputs to TGeomPointInst were objects, and therefore there wasn't any optimization that numpy could do?

Converting the input elements to their proper data types helped, and also using list comprehension instead of .apply was an improvement.

Is it possible to construct TGeomPointInst in such a manner that can use np.vectorize inputs? It's not critical, but it would be nice.

[Diviloper]

Hi @wendellwt, sorry for the delay.
We looked a bit into this when working on the integration with MovingPandas.
We added a factory function to TPointSeq called from_arrays to create a new sequence from three (or four) different arrays : t, x, y and optionally z.
However, there was not a significant gain in it and we moved to other more pressing functionalities.
It is something that we will look into in the future, but as of now, the way you mention (comprehension lists) is probably the fastest.