Parse b3dm file content

We can use python with py3dtiles to parse the information in a b3dm file, for example:

from pathlib import Path
from py3dtiles.tileset.content import B3dm, read_binary_tile_content

filename = Path('./test.b3dm')
b3dm = read_binary_tile_content(filename)

b3dm_header = b3dm.header
print(f"\nb3dm_header.magic_value:\n{b3dm_header.magic_value}")
print(f"\nb3dm_header.version:\n{b3dm_header.version}")

print(f"\nImages in b3dm:")
for image in b3dm.body.gltf.images:
    print(image)

Using Conda behind firewall

 To use conda behind firewall, in Windows:

1) Edit the .condarc

channels:

- defaults

proxy_servers:

  http: <proxy_host>:8080

  https: <proxy_host>:8080

ssl_verify: False

NOTE: Beware doing something like this:

http: http://proxyhost:8080

https: http://proxyhost:8080

Don't add "http://" above, it's wrong!

2) Using pip (need to trust the hosts)

pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org pandas

Some other better methods exist, the above is just one of the options that works.

Extract PDF content (from CHP) with Python

import re

import sys

import urllib.request

import pdfplumber

import pandas as pd

if (len(sys.argv) < 2):

    print("\n\nSyntax: python extract_pdf.py date_string (e.g. 20220124) \n\n")

    exit()

def main():    

    date_string  = sys.argv[1] # e.g. 20220124

    pdf_file_name = f"ctn_{date_string}.pdf" # PDF from CHP: https://www.chp.gov.hk/files/pdf/ctn_20220124.pdf

    #download_pdf( pdf_file_name )

    extract_pdf( pdf_file_name )

def extract_pdf( pdf_file_name ):

    with pdfplumber.open( f"./pdf/{pdf_file_name}" ) as pdf:

        for page in pdf.pages:

            #print(page)

            for table in page.extract_tables():

                df = pd.DataFrame(table[1:], columns=table[0])

                for index, row in df.iterrows():

                    if (isinstance(row[0], str) and len(row[0])>0):

                        rowid = row[0].replace(".","")

                        title = row[1].split("\n")

                        for i in range(0,len(title)):                            

                            if (re.search(u'[\u4e00-\u9fff]', title[i]) is None):

                                title[i] = ""

                        print(rowid, "".join(title))

def download_pdf ( pdf_file_name ) :

    pdfFile = urllib.request.urlopen(f"https://www.chp.gov.hk/files/pdf/{pdf_file_name}")

    file = open(f"./pdf/{pdf_file_name}", "wb")

    file.write(pdfFile.read())

    file.close()

main()

Display cx_oracle error indicating which rows are affected

cursor.executemany("SQL EXEC STATEMENT", data, batcherrors=True)

for error in cursor.getbatcherrors():

    print("Error", error.message, "at row offset", error.offset) 

Tips on GeoPandas

According to the documentation in GeoPandas, the following codes could read a local GeoJSON file and import to a PostGIS DB:

import geopandas

from sqlalchemy import create_engine

path_to_data = "./my_file.geojson"

gdf = geopandas.read_file(path_to_data)

db_connection_url = "postgres://user:pwd@localhost:5432/db";

engine = create_engine(db_connection_url)

gdf.to_postgis("my_file_table", con=engine)

I found the following tips:

• "postgres://" has to be changed to "postgresql://". 
• Package "psycopg2" is required together with "sqlalchemy" (Details could be found in DBAPI support section in PostgreSQL documentation)
• Package "GeoAlchemy2" is required together with "sqlalchemy"

Enjoy!

About CKAN installation from package

 In lubuntu 20.04, we can install CKAN from package via the following command:

# dpkg -i python-ckan_2.9-py3-focal_amd64.deb

The system fails with the following error:

ModuleNotFoundError: No module named 'distutils.core'

This error is caused by missing some essential Python modules, to solve this problem, install the 'python3-distutils' package by:

sudo apt-get install python3-distutils

 

Simple Flask sample

Just wanna to create a very simple Flask application in a single python source file:

simple.py

from flask import Flask

application = Flask("simple")

@application.route("/")

def index():

return "Hello World"

To run this Flask application, run the following:

• export FLASK_APP=simple.py
• flask run

If you want to put this application in a Docker container, you need to bind to 0.0.0.0:

• flask run --host 0.0.0.0

Now, navigate "http://localhost:5000" to see the word "Hello World" displayed on the browser.

Handling table names with underscore ("_") in flask_sqlalchemy / Python

When you create table class using flask_sqlalchemy (the Flask extension for SQLAlchemy), it can maps the modal class to a proper table automatically. This is cool but if you have a table name with underscore (e.g. MY_SAMPLE_TABLE) and you name your model class with the underscore ('_') as usual, you will find SQLAlchemy will map your table name as something the following:

"MY__SAMPLE__TABLE"

Each single underscore (_) with becomes a double underscores (__). This is annoying and you might believe this is a bug, but in fact you can resolve this issues by naming your table class using the PEP8 naming convention, and named your class as:

class MySampleTable(db.Model):
     field_id = db.Column(db.Integer, primary_key = True)
     field_data = db.Column(db.String(200))

The the SQLAlchemy will map the MySampleTable class to table name as "MY_SAMPLE_TABLE" correctly.

For more information about PEP8, click here for details.

Connecting Oracle with Python via cx_Oracle extension

Installation Prerequisites

• Python 2.7, 3.5 or above
• Oracle Client / Oracle Instant Client "Basic" or "Basic Light"
• libaio / libaio1 package

Installation Reference

• cx_Oracle
• Installation guide

Sample Python Code

import cx_Oracle

# Make DSN address. Real values for host address, port and service name were replaced by '<>'.
dsn = cx_Oracle.makedsn('<host_address>', '<port>', service_name='<service_name>')

# Connect to DSN
orcl = cx_Oracle.connect(user='<username>', password='<password>', dsn=dsn)

# Get cursor
cursor = orcl.cursor()

# Execute SQL statement
cursor.execute('<SQL SELECT STATEMENT>')

# Fetch result
data = cursor.fetchone()

Image bytes to np array in Python

For reference:

# data - image bytes

im = Image.open(BytesIO(data))
I =  numpy.asarray(im)
print("I type:", type(I), "I shape:", I.shape)

>> I type: <class 'numpy.ndarray'> I shape: (512, 512, 3)

Conflict between pandas 0.25.0 and pandas-datareader 0.7.0

For Python users, there is a conflict between "pandas 0.25.0" and "pandas-datareader 0.7.0".

If you do an "from pandas_datareader import data", you will got a conflict:

    "ImportError: cannot import name StringIO from pandas.compat"

To get rid of this issue, use "pandas 0.24.2" + "pandas-datareader 0.7.0" instead.