Initial commit.

This commit is contained in:
Mikaël Capelle 2018-02-24 22:43:28 +01:00
commit 02008cb6bc
77 changed files with 6051 additions and 0 deletions

10
.gitignore vendored Normal file
View File

@ -0,0 +1,10 @@
plugins/osm2graph.jar
outputs
inputs/land-polygons-split-4326
*.pbf
*.prj
*.osm
*.shx
*.shp
*.poly
*.dbf

6
.gitmodules vendored Normal file
View File

@ -0,0 +1,6 @@
[submodule "plugins/OSM2Graph"]
path = plugins/OSM2Graph
url = https://github.com/Holt59/OSM2Graph.git
[submodule "inputs/mapsforge-creator"]
path = inputs/mapsforge-creator
url = https://github.com/mapsforge/mapsforge-creator.git

418
.osmosis/changes.txt Normal file
View File

@ -0,0 +1,418 @@
0.46
Require Java 8 as a minimum.
Leverage Java 7 Autocloseable functionality and remove Releasable interface.
Apply minor code cleanups contributed by devfactory.com.
Add tests for pgsimple tasks.
Introduce docker-based build environment.
Introduce Travis CI testing.
Add support for reading protobuf from InputStream. Useful when using Osmosis as a library.
Upgrade to protobuf 3.
Make compression optional in EntitySorter.
0.45
Add postgresSchema option to pgsnapshot tasks.
Verify XML parser on startup to verify support for multi-char unicode characters.
Upgrade to latest version of Netty.
Various dependency version updates.
Fix incorrect plexus classworlds version in Windows launch script.
Upgrade to PostGIS 2.x library.
Add tests for pgsimple database.
0.44
Improve default settings and performance of pgsnapshot load scripts.
Fix remaining 64-bit id tracking problems.
Increase maximum way node count from 16-bit to 32-bit int.
Full null pointer exception in --merge-replication-files.
Fix javadoc errors on JDK 1.8.
Support spaces in installation path on Windows.
Set User-Agent header on all HTTP requests.
Support importing negative ids to apidb.
Fix one-off error in current way/relation loading.
Fix query for retrieving way nodes and relation members in pgsnapshot module.
Prevent reading duplicate way nodes in pgsnapshot module.
Fix deprecation warnings caused by upgrading to newer dependencies.
Upgrade to protobuf 2.6.0.
Fix replicate_osm_file.sh to return the correct exit status, not always -1.
Add scripts to build a docker image for testing database tasks.
Update build to download Gradle via https.
Fix broken console logging in replication lag reader causing a crash.
Update default replication URLs to match current planet server layout.
Add IntelliJ IDE support.
0.43.1
Fix the version number displayed on startup to be up to date.
Remove RELEASE suffix from version number of release builds.
0.43
Update command line help to point to the correct wiki location.
Remove idTrackerType arguments to filtering tasks, only the Dynamic option is now available.
Fix the --write-apidb task to use 64-bit ids.
Upgrade to version 1.4 of the Gradle build tool.
Enhance the build scripts to support publishing to Sonatype OSS Repository and Maven Central.
Rename all projects to use an "osmosis-" prefix.
Included a copy of the PBF OSM Binary project in the source tree, and eliminated the pre-compiled osmbin.jar.
Remove pbfmarshall project due to the same functionality being provided by the osm-binary project.
Removed the internal Ivy repository due to all dependencies now being available on Maven Central.
Rename --read-empty short name --re to --rem to avoid clash with --report-entity.
Rename --read-empty-change short name --rec to --remc for consistency with --read-empty.
0.42
Fix PostgreSQL timestamp bugs in apidb replication logic.
Fix replication file merging boundary corrections. Occurs when catching up after outages.
Replication logic correctly honours the max timestamp parameter.
Prevent replication file downloader from reading beyond maximum available replication interval.
Prevent replication file downloader from stalling if interval is too long.
Improve error reporting when an unknown global option is specified.
Disable automatic state.txt creation for --read-replication-interval.
Add --tag-transform plugin and task.
Reduce number of file handles consumed by file-based sorting.
Make the default id tracker Dynamic for --used-node and --used-way.
Use Gradle for the automated build scripts instead of Ant/Ivy.
Fix PostgreSQL ident authentication.
Remove obsolete debian build scripts.
Eliminate use of deprecated Spring SimpleJdbcTemplate.
Improve handling of invalid geometries in --pgsql-xxx tasks.
Default keepInvalidWays option on --pgsql-xxx tasks to true.
Enable keepInvalidWays functionality for --pgsql-xxx replication.
Fix pgsnapshot COPY load script to use ST_ prefix for all PostGIS functions.
0.41
All entities can now pass metadata through the pipeline. XML tasks can add these as additional attributes on output.
Added --convert-change-to-full-history task.
Added "initialize" support to all tasks to allow pre-processing setup to occur.
Added a number of unit tests to increase test coverage, and reduced use of duplicated support classes.
Added support for the XML bounds element, but still support old bound element.
Added configurable log message prefix to the output of --log-progress-* tasks.
Added completion step timings for --log-progress-* tasks.
Added continuous replication looping support to apidb replication functionality.
Added --write-replication task.
Enhanced --replicate-apidb to send output through the pipeline to tasks like --write-replication instead of being an all-in-one task.
Added --send-replication-sequence task which notifies listeners when new replication data has been written.
Added --send-replication-data task which provides streaming replication data to listeners.
Added --receive-replication task which consumes data sent by --serve-replication-data.
Added --replication-to-change task which converts a replication pipeline stream to a normal change stream.
Added keyValueListFile option to --node-key-value and --way-key-value tasks.
Added JaCoCo code coverage tool to the build process.
Updated ant scripts to use build, checkstyle, test and all targets across all projects.
Update pgsnapshot and pgsimple tasks to no longer use PostGIS deprecated function names.
Added --read-pbf-fast PBF reader which utilises multi-threading to improve performance.
Enhanced XML tasks to handle missing lat/lon attributes on deleted entities.
0.40.1
Fix the classworlds version in the Windows launcher.
0.40
Fixed base64 encoding of user credentials sent by --upload-xml-change.
Added support for merging Bound objects.
Modified entity object creation to require use of CommonEntityData to simplify constructors.
Added metatag facility to all entities to allow custom data to be passed through the pipeline.
Added read timeout to replication retrieval code to prevent it blocking forever.
Improved JPF loading code to avoid NullPointerException if encountering malformed plugins.
Added ability to set input buffer size for all tasks using an input buffer.
Relaxed Bound argument validation to allow points on all edges of the map to be included in bounding boxes.
Modified PostGIS function calls to be PostGIS 2.0 compatible.
Updated build scripts to support GIT.
Do not emit bound entities when reading PBF files with no bbox information set.
Add --compute-bounding-box and --set-bounding-box tasks for manipulating Bound entities.
Added total object count display to --report-entity task.
Updated all jar dependencies with exception of Xerces to latest versions.
Modified project layouts to match Maven standard layout.
Enhanced tests to dynamically generate test data instead of relying on ant-based data generation.
Updated apidb tasks to support new rails3-based apidb schema (rename of id columns).
Fixed name of --log-progress-change-0.6 task. Was registered as --log-change-progress-0.6.
Modified Ivy to symlink to jar files on supported platforms rather copy jars into projects.
0.39
Added the pgsimp tasks which are the old 0.36 version pgsql tasks supporting the old style separate tags tables.
Added a new "cascadingRelations" flag to --bb and --bp which is less costly than "completeRelations" and makes sure that parent relations are emitted when a child relation of theirs is emitted even if that child happens to come after the parent in the stream. The list of relations on output may appear in a different order than on input.
Fixed connection failure error handling in pgsql tasks to display the correct error message instead of a NullPointerException.
Fixed serialisation logic to correctly identify and use the OsmUser.NONE singleton object.
Fixed checkstyle errors in the PBF code.
Renamed the PBF project to "pbf" instead of "binary".
Improved performance in the DataPostbox class used for inter-thread messaging.
Fixed PBF code to allow an empty file to be written.
Updated PBF library dependency from version 1.0 to 1.1.
Updated default PBF filename from dump.osmbin to dump.osm.pbf.
Fixed PBF code to write a header block even if no bound entity exists.
Added java code for PostgreSQL HStore support instead of only including a pre-compiled jar.
Added a new "auto" option for the compressionMethod argument of XML tasks.
Fixed the windows launch batch script to support installation to a different drive than the drive of the working directory.
Fixed the UNIX launch shell script to support spaces in argument values.
Enhanced --used-node to include nodes referenced by ways and relations, instead of just ways.
Enhanced serialisation logic to support negative version numbers. Note this increases the space used by serialisation slightly.
Added a --used-way task.
0.38
Updated internal storage to use long instead of int identifiers to remove 2^31 node id limit.
Updated apidb support to the latest schema 20100910084426.
Fixed the --write-pbf short option --wb to point to the writer implementation instead of the reader.
Added additional quoting to commands in the osmosis.bat launcher to fix issues with spaces in the installation path.
Modified plugin loader code to support multiple osmosis-plugins.conf files to allow each plugin to provide its own.
0.37
Updated the pgsql schema (now version 6) to move all tags into hstore columns, add a ways.nodes column and CLUSTER the nodes and ways tables. Significant performance increases.
Fixed a bug in --bounding-box and --bounding-polygon preventing use of clipIncompleteEntities after a --tee task.
Added --read-pbf and --write-pbf tasks supporting the new Google Protocol Buffer based binary data format.
Added a new --flatten task for removing multiple versions of single entities from an entity stream.
Fixed bugs in the --bounding-box and --bounding-polygon idTrackerType=Dynamic option.
Enhanced the apidb replication read queries by adding temp table primary keys to fix performance regressions with PostgreSQL 8.4.
0.36
Removed 0.5 support.
Added the --fast-write-pgsql task which uses PostgreSQL COPY commands to insert data.
Added the CompactTempFile node location store type which allows efficient pgsql geometry building for small data sets.
All write-pgsql tasks now default to the CompactTempFile node location type store instead of in memory.
Fixed regression in tar builds to make the launch scripts executable again.
Updated the pgsql schema (now version 5) to add a primary key to the relation_members table to allow more efficient updates.
Updated apidb support to the latest schema 20100513171259.
Updated pgsql dataset bounding box queries to force more efficient query plans by disabling inefficient join types.
Added a new dynamic idTrackerType implementation selected by default to the area filtering tasks to simplify usage.
Added the --read-replication-lag task which indicates how far behind the server a local replication is.
0.35
Registered --fast-read-xml as a task instead of requiring --fast-read-xml-0.6 to be specified.
Fixed the completeRelations option on the --bounding-box and --bounding-polygon tasks to work as expected and documented.
Updated the --fast-read-xml task to ignore changeset elements.
Various enhancements to the Ivy processing including publishing of all artefacts.
Fixed apidb tasks to correctly select the MySQL drivers instead of always using PostgreSQL drivers.
Fixed the osmosis.bat launcher to handle spaces in the installation path.
0.34
Fixed the --sort and --sort-change tasks to use 0.6 task implementations instead of the older 0.5 implementations.
Fixed the archive distributions to include the config/plexus.conf file, and include the launcher scripts in the tar version.
Various cleanups of archive distributions to remove unnecessary files and minimise size.
0.33
Use Plexus Classworlds to simplify launcher scripts.
Switch from org.apache.tools.bzip2 to commons-compress (this project is still alive).
Updated the --apply-change task to detect full history diffs and abort.
Enabled cascading deletes in the --truncate-apidb task to minimise foreign key violations on new schema updates.
Updated all PostgreSQL database tasks to support IDENT authentication.
Removed local postgis distribution since it is available on repo1.maven.org now.
0.32
Change to Public Domain licence.
Re-factored all entity classes to use a CommonEntityData class to avoid duplication of code and simplify code re-use.
Significant re-factoring of many apidb tasks to use Spring JDBC Template functionality.
Added a --replicate-apidb task for extracting replication files based on PostgreSQL transaction ids.
Fixed PostgreSQL apidb COPY files to include changeset id.
Fixed the Windows launcher batch file to contain all libraries.
Added a --read-replication-interval task.
Updated XML writing tasks to ignore UTF-8 characters not supported by XML.
Included new Xerces XML library to provide proper support for UTF-16 surrogate pairs.
Added a --merge-replication-files task.
Added a --way-key task.
Added a --simplify-change task.
Added pom.xml for maven artifacts
0.31
Added a --remove-tags task.
Added JPF plugin support.
Updated the build scripts to use Ivy dependency management.
Added mutable entity support.
Added writeable dataset support.
Added apidb tasks that support both mysql and postgresql api databases.
Added a new --append-change task for combining full history changesets.
0.30
Added 0.6 support.
Enhanced the pgsql tasks and schema to support an optional "action" table.
Enhanced the pgsql schema to introduce a stored procedure called during every changeset application.
Enhanced the pgsql schema to include optional bbox and linestring columns on the way table.
Added ability to specify slipping map zoom and coordinates as arguments to the bounding box filter.
Added an "inMemoryBbox" to the pgsql writer task allowing it to calculate bounding boxes using a memory mapped file.
Added an automatic VACUUM ANALYZE of the database for pgsql writer and truncation tasks.
Added migration tasks --migrate and --migrate-change allowing 0.5 data to be converted to 0.6 format.
Added forceUtf8 and profileSql options to all MySQL tasks.
Added a --tag-sort task allowing tags within entities to be sorted alphabetically (0.6 only).
Enhanced the buffer tasks to chunk objects passed (chunks are 1/4 of max buffer size) to minimise thread synchronisation (0.6 only).
Dropped the default buffer size from 100 to 20 which provides slightly improved performance in most cases (0.6 only).
Added the ability to generate full history changesets, as opposed to changesets with a single change per entity (0.6 only).
Added a new allowIncorrectSchemaVersion option to most database tasks.
Added new clipIncompleteEntities option to area filter tasks. It defaults to false which changes existing behaviour.
0.29
Fixed the --write-pgsql-simple-dump task to properly escape '\' characters.
Updated the pgsql simple tasks to use an SRID of 4326.
Updated build script to include a tar file distribution preserving launch script execute permissions.
Enhanced the Entity class to allow dates to be represented as strings to avoid parsing overhead. Updated xml tasks to utilise this feature.
Updated the pgsql simple tasks to use the new database schema which is more similar to the one used in the central MySQL
Removed the old postgresql tasks.
Removed "simple" from the names of the simple pgsql tasks.
Added plugin support allowing new tasks in external jar files to be registered with osmosis.
Added command line checks to ensure all task and option arguments are recognised, any misspelt arguments will now raise an error.
Removed all static methods from the task registrar and task manager factory classes to avoid global data and simplify testing.
Added the --node-key and --node-key-value tasks.
0.28
Added a maximum file download count feature to the change downloader to limit processing overhead when synchronising long time intervals.
Fixed a major error in the TaskRegistrar from 0.27 where multiple tasks had the same name preventing osmosis from launching.
0.27
Added change downloader tasks providing the basis for automatic synchronisation of data sets.
0.26
Removed the bdb dataset implementation.
Fixed an incorrect constant bug in the mysql tile calculator causing incorrect tile values to be generated.
0.25
Added a new osmosis-extract-mysql application for extracting regular changesets from mysql.
Removed use of String.isEmpty to minimise Java5 incompatibility.
Modified the default pipe connectivity functionality to use stack-based connectivity instead of queue-based connectivity.
Added a populateCurrentTables option to the --write-mysql-change task to match the --write-mysql task.
Added validation code to the merge tasks to verify that input data is sorted.
Enhanced serialisation code to eliminate class identification information in the stream where single data types are being stored.
Fixed a bug in the ListIdTracker class where unsorted ids were not being sorted correctly.
Command line applications now returns status codes, 0 for success, non-zero for failure.
Fixed the --read-api short name to be --ra instead of --wa.
Updated store implementations to use buffered io providing significant performance improvements.
Added a new osmosis-test-users-for-utf8 application for verifying the data integrity of the production user table.
Added "bound" support to the pipeline.
Added "-" alias for stdin/stdout to file reading/writing tasks for Cygwin & Windows
Added a new dataset task --write-bdb for writing to a Berkeley DB implementation of a dataset.
Added a new dataset task --read-bdb for reading from a Berkeley DB implementation of a dataset.
Added a new dataset task --write-customdb for writing to a custom implementation of a dataset.
Added a new dataset task --read-customdb for reading from a custom implementation of a dataset.
Added a new dataset task --write-pgsql-simple for writing to a PostgreSQL implementation of a dataset.
Added a new dataset task --write-pgsql-simple-dump for writing "COPY" load files for a PostgreSQL implementation of a dataset.
Added a new dataset task --truncate-pgsql-simple for truncating all PostgreSQL dataset tables.
Added a new dataset task --read-pgsql-simple for reading from a PostgreSQL implementation of a dataset.
Added a new dataset task --dataset-bounding-box for extracting a bounding box from a dataset.
Added a new dataset task --dataset-dump for dumping the entire contents of a dataset.
Added new --used-node and --way-key-value tasks for filtering ways and their associated nodes with specific tags.
0.24
Fixed a bug in the XML node writing code where it would write a ',' character instead of '.' as the decimal separator in some locales.
0.23
Added support for a custom decoder when writing xml files to workaround the current production utf8 double encoding issue.
Added short task names to shorten command line (eg. --write-xml becomes --wx).
Fixed a node changeset bug where nodes created at the end of the interval were being excluded from the query.
Added support for default task arguments requiring no name and added them for commonly used tasks requiring a single argument.
0.22
Updated mysql writing tasks to check if a timestamp is set and raise a meaningful error indicating where the error occurred.
Modified mysql tasks to use the default character encoding for database connections to avoid suspected double encoding with the production database.
Fixed the Entity serialisation to cater for null user ids and timestamps.
Removed all 0.4 tasks.
Added a new IdList id tracker implementation to be used internally by area filtering tasks to reduce memory usage on small'ish areas. Old BitSet implementation is still selectable if large bounding boxes are required.
Added automatic compression method detection for all xml file tasks to avoid need to use compressionMethod argument.
0.21
Introduced a custom serialisation mechanism to improve compressed temporary file performance by approximately 5 times.
Added a new RandomAccessObjectStore class which can be used as the basis for smarter tasks requiring access to data multiple times.
Added completeWays and completeRelations arguments to the area filtering tasks to provide more control over the filtering process.
Added new --tee and --tee-change tasks for sending data to multiple output destinations.
0.20
Fixed the integrity reporter task so that it writes new lines after each record.
Relaxed validation checks in the --apply-change task so that it can apply changes to planets already containing data without aborting.
0.19
Added a new --integrity-reporter task to detect referential integrity issues between entity types.
Updated the default file names for report tasks.
Added a new --log-progress task to provide progress information to the jdk log during processing.
Added a new --log-change-progress task to provide change progress information to the jdk log during processing.
Fixed a number of bugs in the area extraction tasks.
Updated area extraction tasks to support negative identifiers used on files that are ready for upload.
Fixed bug in the ChangeWriter where the node insert SQL commands were missing tile parameters.
0.18
Improved error messages when errors are returned from the api.
Added support for new shorter date format excluding millisecond information.
Updated mysql writing code to support the new 0.5 schema version and fixed bugs in node insertion sql.
Added support for quiet and verbose command line options.
Modified command line arguments to default to using 0.5 tasks.
0.17
Added schema version checking to all MySQL tasks.
Updated xml reading tasks to provide line number information when parse errors occur.
Modified xml writing tasks to include the osmosis version number in the generator attribute.
Updated mysql tasks to support the new tile indexing and fixed precision lat/lon.
0.16
Fixed a major bug in UTF-8 handling when writing xml files. All files were being written in the default character set.
0.15
Added full 0.5 support.
Updated 0.4 code to the 005 schema.
0.14
Fixed a bug in the area filtering tasks where the original way was being passed to the destination instead of the modified way with only the available segments populated.
Fixed a bug in the way comparison code where changes in the ordering of segments wouldn't be picked up.
Fixed a bug in date parsing where the month was always being parsed a month in advance.
Added the ability to load database login credentials from a properties file.
Modified xml file writing to use 2 space indenting instead of 4.
0.13
Modified the --bounding-polygon task so it can read files containing blank lines.
Added the ability to dump mysql "current" tables.
Modified xml output to only write latitude and longitude to 7 decimal places.
0.12
Added a --read-api task for retrieving data from the central server.
Added "user" support to the core data types, and all relevant tasks except database writing tasks.
Added a --bounding-polygon task for filtering data by an area.
Added a --report task which will produce user statistics for a data set.
0.11
Modified mysql snapshot task to read entire contents of table and process data in code in attempt to improve dump times.
Added gzip and bzip2 support to xml tasks.
Modified all database snapshot tasks to include entities with a timestamp equalling the snapshot time.
Modified all database changeset tasks to not include the begin time and to include the end time.
0.10
Added JOSM-like date handling as a fallback if the standard xml date format isn't found.
Added a custom date parser for the special case of GMT times which this tool always produces (approx 10x speedup over JOSM code and 100x speedup over standard JDK for GMT dates).
0.9
Fixed the name of the main class attribute in the jar manifest.
Fixed a potential bug in the change application code where a validation step may have erroneously thrown errors.
Added merge tasks to combine multiple entity or change files together.
0.8
Modified node and segment readers to avoid returning duplicate records where multiple records with identical timestamps exist.
Moved main osmosis code into a "core" sub-package allowing new tools to be added.
Fixed timeouts in mysql dumping of ways by using temporary files for query results.
Refactored all temporary file handling to use common persistent iterator functionality.
Fixed some classes not marked as Serializable.
Updated change application to use less strict validation to handle "re-creates" of entities.
Updated xml writing tasks to always write in UTC format.
0.7
Made a number of changes to the mysql dump task to improve performance and reduce the likelihood of timeouts.
0.6
Modified ChangeDeriver task to set the timestamp of deleted records to the current time.
Fixed all uses of java.sql.Timestamp to be converted to java.util.Date explicitly instead of utilising inheritance.
Fixed bug in change writer where timestamps weren't being written for segments.
Added UserIdManager functionality which creates a system "Osmosis" user in the database to assign all changes to.
0.5
Modified command line date format to avoid use of spaces.
0.4
Modified mysql dump task to produce consistent snapshots.
Added the ability to skip "current" tables when importing to mysql.
Optimised mysql change reading queries.
0.3
Added mysql table truncator task.
Updated segment xml processing code to handle timestamps.
Made xml date processing during xml reading optional to improve performance.
Added buffer tasks to allow pipeline processing to be split across multiple threads.
Change reading and writing from a mysql database is now working, at least for simple test cases.
0.2
Added Main-Class feature to the jar manifest simplifying program launch.
Added a shell launch script to simplify running on unix/linux.
Added a readme with basic installation instructions.
Added xml date parsing support.
Fixed database dump code to read the "current" tables.
Added a task for reading changes from a database.
Added a task for writing changes to a database.
Added a task for truncating database tables.
Improved mysql import performance by disabling indexes and locking tables.
0.1
Initial version.

16
.osmosis/copying.txt Normal file
View File

@ -0,0 +1,16 @@
Osmosis consists of all files in this archive with the exception of the
third party libraries in the lib sub-directory, and the osmosis-osm-binary
library which is a re-packaged version of a third-party library.
Osmosis is placed into the public domain and where this is not legally
possible everybody is granted a perpetual, irrevocable license to use
this work for any purpose whatsoever.
DISCLAIMERS
By making Osmosis publicly available, it is hoped that users will find the
software useful. However:
* Osmosis comes without any warranty, to the extent permitted by applicable
law.
* Unless required by applicable law, no liability will be accepted by
the authors and distributors of this software for any damages caused
as a result of its use.

42
.osmosis/readme.txt Normal file
View File

@ -0,0 +1,42 @@
INSTALLATION
Unzip the distribution in the location of your choice.
On unix/linux systems, make the bin/osmosis script executable (ie. chmod u+x osmosis).
If desired, create a symbolic link to the osmosis script somewhere on your path (eg. ln -s appdir/bin/osmosis ~/bin/osmosis).
CONFIGURATION
Common command line options can be specified in configuration files.
On Linux, the file can reside in the following locations with later files overriding earlier files:
/etc/osmosis
$HOME/.osmosis
On Windows, the file can reside in the following locations with later files overriding earlier files:
%ALLUSERSPROFILE%\osmosis.bat
%USERPROFILE%\osmosis.bat
The following variables can be defined in these files:
JAVACMD - The java command to be invoked. Default is "java".
JAVACMD_OPTIONS - The java jvm options to apply (eg. -Xmx512M).
OSMOSIS_OPTIONS - The osmosis options to apply (eg. -q or -plugin MyPluginLoaderClass).
COMPILATION
To perform a complete osmosis rebuild, the following command may be run from the osmosis root directory.
ant all
The "all" ant target performs all steps including creation of new distribution files, checkstyle analysis and unit tests.
Sometimes old files can be left hanging around causing problems. It may be necessary to run the following command
to clean up any old files.
ant clean
If you wish to rebuild all artefacts without running unit tests, the following command may be used.
ant publish
HELP
Osmosis documentation is available at:
http://wiki.openstreetmap.org/index.php/Osmosis
Please ask any questions, report any issues, or suggest enhancements using the Open Street Map talk or development mailing lists.

94
bin/osmosis Executable file
View File

@ -0,0 +1,94 @@
#!/bin/sh
# Config files can define several variables used throughout this script.
# JAVACMD - The java command to launch osmosis.
# JAVACMD_OPTIONS - The options to append to the java command, typically used to modify jvm settings such as max memory.
# OSMOSIS_OPTIONS - The options to apply to all osmosis invocations, typically used to add plugins or make quiet operation the default.
if [ -f /etc/osmosis ] ; then
. /etc/osmosis
fi
if [ -f "$HOME/.osmosis" ] ; then
. "$HOME/.osmosis"
fi
if [ -z "$JAVACMD" ] ; then
# No JAVACMD provided in osmosis config files, therefore default to java
JAVACMD=java
fi
## resolve links - $0 may be a link to application
PRG="$0"
# if started without absolute path, but from PATH environment
if [ ! -s "$PRG" ] ; then
PRG=`which $PRG`
fi
# need this for relative symlinks
while [ -h "$PRG" ] ; do
ls=`ls -ld "$PRG"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '/.*' > /dev/null; then
PRG="$link"
else
PRG="`dirname "$PRG"`/$link"
fi
done
if [ "x$1x" = "xx" ] || echo "$@" | grep -q -e '--help' ; then
cat <<EOF
osmosis
Example Usage
Import a planet file into a local PostgreSQL database.
osmosis --read-xml file=~/osm/planbet/planet.osm --write-apidb host="x" database="x" user="x" password="x"
Export a planet file from a local PostgreSQL database.
osmosis --read-apidb host="x" database="x" user="x" password="x" --write-xml file="planet.osm"
Derive a change set between two planet files.
osmosis --read-xml file="planet2.osm" --read-xml file="planet1.osm" --derive-change --write-xml-change file="planetdiff-1-2.osc"
Derive a change set between a planet file and a database.
osmosis --read-mysql host="x" database="x" user="x" password="x" --read-xml file="planet1.osm" --derive-change --write-xml-change file="planetdiff-1-2.osc"
Apply a change set to a planet file.
osmosis --read-xml-change file="planetdiff-1-2.osc" --read-xml file="planet1.osm" --apply-change --write-xml file="planet2.osm"
Sort the contents of a planet file.
osmosis --read-xml file="data.osm" --sort type="TypeThenId" --write-xml file="data-sorted.osm"
The above examples make use of the default pipe connection feature, however a simple read and write planet file command line could be written in two ways. The first example uses default pipe connection, the second explicitly connects the two components using a pipe named "mypipe". The default pipe connection will always work so long as each task is specified in the correct order.
osmosis --read-xml file="planetin.osm" --write-xml file="planetout.osm"
osmosis --read-xml file="planetin.osm" outPipe.0="mypipe" --write-xml file="planetout.osm" inPipe.0="mypipe"
Full usage details are available at: http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage
EOF
exit 1
fi
# make it fully qualified
saveddir=`pwd`
MYAPP_HOME=`dirname "$PRG"`/..
MYAPP_HOME=`cd "$MYAPP_HOME" && pwd`
cd "$saveddir"
# Build up the classpath of required jar files via classworlds launcher.
MYAPP_CLASSPATH=$MYAPP_HOME/lib/default/plexus-classworlds-*.jar
MAINCLASS=org.codehaus.classworlds.Launcher
EXEC="$JAVACMD $JAVACMD_OPTIONS -cp $MYAPP_CLASSPATH -Dapp.home=$MYAPP_HOME -Dclassworlds.conf=$MYAPP_HOME/config/plexus.conf $MAINCLASS $OSMOSIS_OPTIONS"
exec $EXEC "$@"

54
bin/osmosis-extract-apidb-0.6 Executable file
View File

@ -0,0 +1,54 @@
#!/bin/sh
# Config files can define several variables used throughout this script.
# JAVACMD - The java command to launch osmosis.
# JAVACMD_OPTIONS - The options to append to the java command, typically used to modify jvm settings such as max memory.
if [ -f /etc/osmosis ] ; then
. /etc/osmosis
fi
if [ -f "$HOME/.osmosis" ] ; then
. "$HOME/.osmosis"
fi
if [ -z "$JAVACMD" ] ; then
# No JAVACMD provided in osmosis config files, therefore default to java
JAVACMD=java
fi
## resolve links - $0 may be a link to application
PRG="$0"
# if started without absolute path, but from PATH environment
if [ ! -s "$PRG" ] ; then
PRG=`which $PRG`
fi
# need this for relative symlinks
while [ -h "$PRG" ] ; do
ls=`ls -ld "$PRG"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '/.*' > /dev/null; then
PRG="$link"
else
PRG="`dirname "$PRG"`/$link"
fi
done
# make it fully qualified
saveddir=`pwd`
MYAPP_HOME=`dirname "$PRG"`/..
MYAPP_HOME=`cd "$MYAPP_HOME" && pwd`
cd "$saveddir"
# Build up the classpath of required jar files.
MYAPP_CLASSPATH=$MYAPP_HOME/osmosis.jar:$OSMOSIS_CLASSPATH
for FILE in `ls $MYAPP_HOME/lib/default/`; do
MYAPP_CLASSPATH=$MYAPP_CLASSPATH:$MYAPP_HOME/lib/default/$FILE
done
MAINCLASS=org.openstreetmap.osmosis.extract.apidb.v0_6.OsmosisExtractApiDb
EXEC="$JAVACMD $JAVACMD_OPTIONS -cp $MYAPP_CLASSPATH $MAINCLASS $@"
exec $EXEC

54
bin/osmosis-extract-mysql-0.6 Executable file
View File

@ -0,0 +1,54 @@
#!/bin/sh
# Config files can define several variables used throughout this script.
# JAVACMD - The java command to launch osmosis.
# JAVACMD_OPTIONS - The options to append to the java command, typically used to modify jvm settings such as max memory.
if [ -f /etc/osmosis ] ; then
. /etc/osmosis
fi
if [ -f "$HOME/.osmosis" ] ; then
. "$HOME/.osmosis"
fi
if [ -z "$JAVACMD" ] ; then
# No JAVACMD provided in osmosis config files, therefore default to java
JAVACMD=java
fi
## resolve links - $0 may be a link to application
PRG="$0"
# if started without absolute path, but from PATH environment
if [ ! -s "$PRG" ] ; then
PRG=`which $PRG`
fi
# need this for relative symlinks
while [ -h "$PRG" ] ; do
ls=`ls -ld "$PRG"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '/.*' > /dev/null; then
PRG="$link"
else
PRG="`dirname "$PRG"`/$link"
fi
done
# make it fully qualified
saveddir=`pwd`
MYAPP_HOME=`dirname "$PRG"`/..
MYAPP_HOME=`cd "$MYAPP_HOME" && pwd`
cd "$saveddir"
# Build up the classpath of required jar files.
MYAPP_CLASSPATH=$MYAPP_HOME/osmosis.jar:$OSMOSIS_CLASSPATH
for FILE in `ls $MYAPP_HOME/lib/default/`; do
MYAPP_CLASSPATH=$MYAPP_CLASSPATH:$MYAPP_HOME/lib/default/$FILE
done
MAINCLASS=org.openstreetmap.osmosis.extract.mysql.v0_6.OsmosisExtractMysql
EXEC="$JAVACMD $JAVACMD_OPTIONS -cp $MYAPP_CLASSPATH $MAINCLASS $@"
exec $EXEC

41
bin/osmosis.bat Normal file
View File

@ -0,0 +1,41 @@
@ECHO OFF
REM This is an equivalent Windows batch file to complement the unix shell script
REM Corresponding lines from the shell script are printed before the matching batch file commands
REM # Config files can define several variables used throughout this script.
REM # JAVACMD - The java command to launch osmosis.
REM # JAVACMD_OPTIONS - The options to append to the java command, typically used to modify jvm settings such as max memory.
REM # OSMOSIS_OPTIONS - The options to apply to all osmosis invocations, typically used to add plugins or make quiet operation the default.
REM if [ -f /etc/osmosis ] ; then
REM . /etc/osmosis
REM fi
IF EXIST "%ALLUSERSPROFILE%\osmosis.bat" CALL "%ALLUSERSPROFILE%\osmosis.bat"
REM if [ -f "$HOME/.osmosis" ] ; then
REM . "$HOME/.osmosis"
REM fi
IF EXIST "%USERPROFILE%\osmosis.bat" CALL "%USERPROFILE%\osmosis.bat"
REM if [ -z "$JAVACMD" ] ; then
REM # No JAVACMD provided in osmosis config files, therefore default to java
REM JAVACMD=java
REM fi
IF "%JAVACMD%"=="" set JAVACMD=java
REM Set "SAVEDIR" to the current directory
set SAVEDIR=%CD%
set MYAPP_HOME=%~dp0..
REM Now make the MYAPP_HOME path absolute
cd /D %MYAPP_HOME%
set MYAPP_HOME=%CD%
REM Change back to the original directory
cd /D %SAVEDIR%
set MAINCLASS=org.codehaus.classworlds.Launcher
set PLEXUS_CP=%MYAPP_HOME%\lib\default\plexus-classworlds-2.5.2.jar
SET EXEC="%JAVACMD%" %JAVACMD_OPTIONS% -cp "%PLEXUS_CP%" -Dapp.home="%MYAPP_HOME%" -Dclassworlds.conf="%MYAPP_HOME%\config\plexus.conf" %MAINCLASS% %OSMOSIS_OPTIONS% %*
%EXEC%

5
config/plexus.conf Normal file
View File

@ -0,0 +1,5 @@
main is org.openstreetmap.osmosis.core.Osmosis from osmosis.core
[osmosis.core]
load ${app.home}/lib/default/*.jar
load ${app.home}/config

126
generate.sh Executable file
View File

@ -0,0 +1,126 @@
#!/bin/bash
DEFAULT_OUTPUT_FOLDER=outputs
bounding_box=
debug=false
input_file=
output_name=
# FUNCTION help_short()
#
# Print short usage information.
function help_short() {
echo "Usage: $0 [-h|--help] [-d|--debug] [-b|--bounding-box] [-o|--output OUTPUT_NAME] INPUT_FILE"
}
# FUNCTION help_long()
#
# Print detailed usage information.
function help_long() {
help_short
printf "\t-d,--debug\tEnable debug mode.\n"
printf "\t-b,--bounding-box \tSpecify a bounding-box, either via a string or a file.\n"
printf "\t-h,--help\tPrint this help.\n"
}
# FUNCTION error(...)
#
# Print error composed of arguments, then exit.
function error() {
echo "$0: $*" >> /dev/stderr
exit 1
}
# FUNCTION error_and_usage(...)
#
# Print error composed of arguments, followed by help, then exit.
function error_and_usage() {
echo "$0: $*" >> /dev/stderr
help_short
exit 1
}
# FUNCTION warning(...)
#
# Print warning / information composed of arguments.
function warning() {
echo "$0: $*" >> /dev/stderr
}
# FUNCTION run(...)
#
# Execute given command if debug mode is disabled, otherwize
# print it to stderr.
function run() {
$debug && warning $*
$debug || $*
}
# FUNCTION check_input_file()
#
# Check if input file exists and has read access.
function() check_input_file() {
if [ -z "${input_file}" ]; then
error_and_usage "Missing input file."
fi
if [ ! -e "${input_file}" ]; then
error "${input_file} does not exist."
fi
if [ ! -e "${input_file}" ]; then
error "${input_file} does not exist."
fi
if [ ! -r "${input_file}" ]; then
error "${input_file} is not readable."
fi
}
# FUNCTION check_output_file()
#
# Check output file.
function() check_input_file() {
if [ -z "${output_file}" ]; then
output_file=${DEFAULT_OUTPUT_FOLDER}/$(basename -s .pbf ${input_file})
warning "No output file specific, switching to ${output_file}."
fi
}
# Loop for command line arguments, see:
# http://stackoverflow.com/a/14203146/2666289
while [ ! -z "$1" ]
do
key="$1"
case $key in
-b|--bounding-box)
bounding_box="$2"
shift
;;
-d|--debug)
debug=true
;;
-o|--output)
output_name="$2"
shift
;;
-h|--help)
help_long
exit 0
;;
*)
input_file="$key"
;;
esac
shift # past argument or value
done
check_input_file
check_output_file

1
inputs/bbox/insa.bbox Normal file
View File

@ -0,0 +1 @@
bottom=43.563533 left=1.457989 top=43.578851 right=1.477515

View File

@ -0,0 +1 @@
bottom=47.234 left=-3.598 top=47.997 right=-1.648

104
inputs/coastline-fix.sh Executable file
View File

@ -0,0 +1,104 @@
#!/bin/bash
# Mapsforge map creation (with coastlines) - Modified by Mikael
# Osmosis folder
OSMOSIS=/opt/osmosis/bin/osmosis
# Input folder
INPUTS=/opt/osmosis/inputs
# Change to desired region
REGION=bretagne
# Use existing files, but if necessary can download...
# echo "Downloading $1..."
# wget -nv -N -P "$WORK_PATH" http://download.geofabrik.de/$1-latest.osm.pbf || exit 1
# wget -nv -N -P "$WORK_PATH" http://download.geofabrik.de/$1-latest.osm.pbf.md5 || exit 1
# (cd "$WORK_PATH" && exec md5sum -c "$NAME-latest.osm.pbf.md5") || exit 1
# wget -nv -N -P "$WORK_PATH" http://download.geofabrik.de/$1.poly || exit 1
# Bounds
BBOX=$(perl poly2bb.pl "$INPUTS/$REGION/$REGION.poly")
BBOX=(${BBOX//,/ })
BOTTOM=${BBOX[0]}
LEFT=${BBOX[1]}
TOP=${BBOX[2]}
RIGHT=${BBOX[3]}
# Start position
CENTER=$(perl poly2center.pl "$INPUTS/$REGION/$REGION.poly")
CENTER=(${CENTER//,/ })
LAT=${CENTER[0]}
LON=${CENTER[1]}
# Land
ogr2ogr -overwrite -progress -skipfailures -clipsrc $LEFT $BOTTOM $RIGHT $TOP "$INPUTS/$REGION/${REGION}_land.shp" "$INPUTS/land-polygons-split-4326/land_polygons.shp"
python shape2osm.py -l "$INPUTS/$REGION/${REGION}_land" "$INPUTS/$REGION/${REGION}_land.shp"
# Sea
cp sea.osm "$INPUTS/$REGION/"
sed -i "s/\$BOTTOM/$BOTTOM/g" "$INPUTS/$REGION/sea.osm"
sed -i "s/\$LEFT/$LEFT/g" "$INPUTS/$REGION/sea.osm"
sed -i "s/\$TOP/$TOP/g" "$INPUTS/$REGION/sea.osm"
sed -i "s/\$RIGHT/$RIGHT/g" "$INPUTS/$REGION/sea.osm"
CMD="$OSMOSIS --rb file=$INPUTS/$REGION/$REGION-latest.osm.pbf \
--rx file=$INPUTS/$REGION/sea.osm --s --m"
for f in $INPUTS/$REGION/${REGION}_land*.osm; do
CMD="$CMD --rx file=$f --s --m"
done
CMD="$CMD --wb file=$INPUTS/$REGION/${REGION}_merge.pbf omitmetadata=true"
echo $CMD
$CMD
exit 0
# Map
if [ "$SKIP_MAP_CREATION" != "true" ]; then
CMD="$OSMOSIS_HOME/bin/osmosis --rb file=$WORK_PATH/merge.pbf"
[ $MAP_TRANSFORM_FILE ] && CMD="$CMD --tt file=$MAP_TRANSFORM_FILE"
CMD="$CMD --mw file=$WORK_PATH/$NAME.map \
type=$2 \
bbox=$BOTTOM,$LEFT,$TOP,$RIGHT \
map-start-position=$LAT,$LON \
map-start-zoom=8 \
tag-values=$TAG_VALUES \
threads=$THREADS"
[ $3 ] && CMD="$CMD preferred-languages=$3"
[ $MAP_TAG_CONF_FILE ] && CMD="$CMD tag-conf-file=$MAP_TAG_CONF_FILE"
echo $CMD
$CMD || exit 1
# Check map size
if [ -f "$MAPS_PATH/$NAME.map" ]; then
OLD_SIZE=$(wc -c < "$MAPS_PATH/$NAME.map")
NEW_SIZE=$(wc -c < "$WORK_PATH/$NAME.map")
if [ $NEW_SIZE -lt $(($OLD_SIZE * 70 / 100)) ]; then
echo "$WORK_PATH/$NAME.map creation is significantly smaller."
exit 1
fi
fi
mv "$WORK_PATH/$NAME.map" "$MAPS_PATH/$NAME.map"
fi
# POI
if [ "$SKIP_POI_CREATION" != "true" ]; then
CMD="$OSMOSIS_HOME/bin/osmosis --rb file=$WORK_PATH/$NAME-latest.osm.pbf \
--pw file=$WORK_PATH/$NAME.poi"
[ $POI_TAG_CONF_FILE ] && CMD="$CMD tag-conf-file=$POI_TAG_CONF_FILE"
echo $CMD
$CMD || exit 1
mv "$WORK_PATH/$NAME.poi" "$POIS_PATH/$NAME.poi"
fi
# Post-process
rm -rf "$WORK_PATH"

View File

@ -0,0 +1 @@
motorway,trunk,primary,secondary,motorway_link,trunk_link,primary_link,secondary_link,tertiary,residential,unclassified,road,living_street,service,roundabout,coastline

@ -0,0 +1 @@
Subproject commit c0f669e55a6f2fd9412a39d8d9733f7567b54ea8

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

BIN
lib/default/jpf-1.5.jar Normal file

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

1
plugins/OSM2Graph Submodule

@ -0,0 +1 @@
Subproject commit 675fa962d977b377bb5ddf00b8ccd6b920769131

Binary file not shown.

View File

@ -0,0 +1,51 @@
-------------------------------------------------------------------------------
-- The following script creates a new table for the pgsql simple schema for
-- storing full way geometries.
--
-- Author: Ralf
-------------------------------------------------------------------------------
-- drop table if it exists
DROP TABLE IF EXISTS way_geometry;
-- create table
CREATE TABLE way_geometry(
way_id bigint NOT NULL
);
-- add PostGIS geometry column
SELECT AddGeometryColumn('', 'way_geometry', 'geom', 4326, 'GEOMETRY', 2);
-------------------------------------------------------------------------------
-- the following might go into the POST_LOAD_SQL-array in the class "PostgreSqlWriter"??
-------------------------------------------------------------------------------
-- add a linestring for every way (create a polyline)
INSERT INTO way_geometry select id, ( select ST_LineFromMultiPoint( Collect(nodes.geom) ) from nodes
left join way_nodes on nodes.id=way_nodes.node_id where way_nodes.way_id=ways.id ) FROM ways;
-- after creating a line for every way (polyline), we want closed ways to be stored as polygones.
-- So we need to delete the previously created polylines for these ways first.
DELETE FROM way_geometry WHERE way_id IN
( SELECT ways.id FROM ways
WHERE ST_IsClosed( (SELECT ST_LineFromMultiPoint( Collect(n.geom) ) FROM nodes n LEFT JOIN way_nodes wn ON n.id=wn.node_id WHERE ways.id=wn.way_id) )
AND ST_NumPoints( (SELECT ST_LineFromMultiPoint( Collect(n.geom) ) FROM nodes n LEFT JOIN way_nodes wn ON n.id=wn.node_id WHERE ways.id=wn.way_id) ) >= 3
)
;
-- now we need to add the polyline geometry for every closed way
INSERT INTO way_geometry SELECT ways.id,
( SELECT ST_MakePolygon( ST_LineFromMultiPoint(Collect(nodes.geom)) ) FROM nodes
LEFT JOIN way_nodes ON nodes.id=way_nodes.node_id WHERE way_nodes.way_id=ways.id
)
FROM ways
WHERE ST_IsClosed( (SELECT ST_LineFromMultiPoint( Collect(n.geom) ) FROM nodes n LEFT JOIN way_nodes wn ON n.id=wn.node_id WHERE ways.id=wn.way_id) )
AND ST_NumPoints( (SELECT ST_LineFromMultiPoint( Collect(n.geom) ) FROM nodes n LEFT JOIN way_nodes wn ON n.id=wn.node_id WHERE ways.id=wn.way_id) ) >= 3
;
-------------------------------------------------------------------------------
-- create index on way_geometry
CREATE INDEX idx_way_geometry_way_id ON way_geometry USING btree (way_id);
CREATE INDEX idx_way_geometry_geom ON way_geometry USING gist (geom);

4096
script/contrib/apidb_0.6.sql Normal file

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,28 @@
-- This script creates a function and indexes that allow osmosis to efficiently query based on transaction ids.
CREATE OR REPLACE FUNCTION xid_to_int4(t xid)
RETURNS integer AS
$BODY$
DECLARE
tl bigint;
ti int;
BEGIN
tl := t;
IF tl >= 2147483648 THEN
tl := tl - 4294967296;
END IF;
ti := tl;
RETURN ti;
END;
$BODY$
LANGUAGE 'plpgsql' IMMUTABLE STRICT;
DROP INDEX IF EXISTS nodes_xmin_idx;
DROP INDEX IF EXISTS ways_xmin_idx;
DROP INDEX IF EXISTS relations_xmin_idx;
CREATE INDEX nodes_xmin_idx ON nodes USING btree ((xid_to_int4(xmin)));
CREATE INDEX ways_xmin_idx ON ways USING btree ((xid_to_int4(xmin)));
CREATE INDEX relations_xmin_idx ON relations USING btree ((xid_to_int4(xmin)));

2
script/contrib/dump_apidb.sh Executable file
View File

@ -0,0 +1,2 @@
#!/bin/sh
/usr/bin/pg_dump -O -v -f "apidb_0.6.sql" api06

View File

@ -0,0 +1,37 @@
#!/bin/sh
# This script automates the replication of changes into an offline osm file for a specific area of interest. This allows an up-to-date local snapshot of an area to be maintained.
# The name of the replicated file.
OSM_FILE=myfile.osm.gz
# The name of the temp file to create during processing (Note: must have the same extension as OSM_FILE to ensure the same compression method is used.
TEMP_OSM_FILE=tmp.osm.gz
# The directory containing the state associated with the --read-change-interval task previously initiated with the --read-change-interval-init task.
WORKING_DIRECTORY=./
# The bounding box to maintain.
LEFT=-180
BOTTOM=-90
RIGHT=180
TOP=90
# The osmosis command.
CMD="osmosis -q"
# Launch the osmosis process.
$CMD --read-change-interval $WORKING_DIRECTORY --read-xml $OSM_FILE --apply-change --bounding-box left=$LEFT bottom=$BOTTOM right=$RIGHT top=$TOP --write-xml $TEMP_OSM_FILE
STATUS=$?
# Verify that osmosis ran successfully.
if [ "$STATUS" -ne "0" ]; then
echo "Osmosis failed, aborting."
exit $STATUS
fi
mv $TEMP_OSM_FILE $OSM_FILE

14
script/fix_line_endings.sh Executable file
View File

@ -0,0 +1,14 @@
#!/bin/sh
find . -iname "*.java" -exec dos2unix -U '{}' \;
find . -iname "*.xml" -exec dos2unix -U '{}' \;
find . -iname "*.txt" -exec dos2unix -U '{}' \;
find . -iname "*.osm" -exec dos2unix -U '{}' \;
find . -iname "*.osc" -exec dos2unix -U '{}' \;
find . -iname "*.java" -exec svn propset svn:eol-style native '{}' \;
find . -iname "*.xml" -exec svn propset svn:eol-style native '{}' \;
find . -iname "*.txt" -exec svn propset svn:eol-style native '{}' \;
find . -iname "*.osm" -exec svn propset svn:eol-style native '{}' \;
find . -iname "*.osc" -exec svn propset svn:eol-style native '{}' \;

7
script/munin/README Normal file
View File

@ -0,0 +1,7 @@
to activate the munin plugins
- copy "osm_replication_lag" to "/usr/share/munin/plugins"
- make "/usr/share/munin/plugins/osm_replication_lag" executable
- symlink "/usr/share/munin/plugins/osm_replication_lag" to "/etc/munin/plugins"
- copy "osm_replication.conf" to "/etc/munin/plugin-conf.d"
- edit "/etc/munin/plugin-conf.d/osm_replication.conf" and set the workingDirectory
- restart the munin-node

View File

@ -0,0 +1,15 @@
[osm*]
# the osmosis invocation may take some time
timeout 60
# the system user that has access to the working directory, if it'S different
# from "munin"
#user osm
# path to the osmosis binary. if not set, osmosis is assumed to be in $PATH
#env.osmosis /opt/osmosis/bin/osmosis
# working directory of the osmosis replication.
# this must be set to make the munin plugin work
#env.workingDirectory /path/to/state.txt

View File

@ -0,0 +1,35 @@
#!/bin/sh
# -*- sh -*-
# load the munin plugin helper
. $MUNIN_LIBDIR/plugins/plugin.sh
# if no workingDirectory has been configures
if [ ! $workingDirectory ]; then
# exit with an error
echo "no workingDirectory configured" >&2
exit 1
fi
# path to osmosis binary
[ $osmosis ] || osmosis="osmosis"
# configuration section
if [ "$1" = "config" ]; then
echo 'graph_title OSM PostGIS Database Replag'
echo 'graph_args --base 1000'
echo 'graph_vlabel seconds behind main database'
echo 'graph_category osm'
echo 'lag.label replication lag'
echo 'lag.draw LINE'
exit 0
fi
# invoke osmosis to calculate the replication lag
lag=$($osmosis --read-replication-lag workingDirectory="$workingDirectory" 2>/dev/null)
echo "lag.value $lag"
exit 0

View File

@ -0,0 +1,67 @@
-- Drop all primary keys and indexes to improve load speed.
ALTER TABLE nodes DROP CONSTRAINT pk_nodes;
ALTER TABLE ways DROP CONSTRAINT pk_ways;
ALTER TABLE way_nodes DROP CONSTRAINT pk_way_nodes;
ALTER TABLE relations DROP CONSTRAINT pk_relations;
ALTER TABLE relation_members DROP CONSTRAINT pk_relation_members;
DROP INDEX idx_node_tags_node_id;
DROP INDEX idx_nodes_geom;
DROP INDEX idx_way_tags_way_id;
DROP INDEX idx_way_nodes_node_id;
DROP INDEX idx_relation_tags_relation_id;
DROP INDEX idx_ways_bbox;
DROP INDEX idx_ways_linestring;
-- Comment these out if the COPY files include bbox or linestring column values.
SELECT DropGeometryColumn('ways', 'bbox');
SELECT DropGeometryColumn('ways', 'linestring');
-- Import the table data from the data files using the fast COPY method.
\copy users FROM 'users.txt'
\copy nodes FROM 'nodes.txt'
\copy node_tags FROM 'node_tags.txt'
\copy ways FROM 'ways.txt'
\copy way_tags FROM 'way_tags.txt'
\copy way_nodes FROM 'way_nodes.txt'
\copy relations FROM 'relations.txt'
\copy relation_tags FROM 'relation_tags.txt'
\copy relation_members FROM 'relation_members.txt'
-- Add the primary keys and indexes back again (except the way bbox index).
ALTER TABLE ONLY nodes ADD CONSTRAINT pk_nodes PRIMARY KEY (id);
ALTER TABLE ONLY ways ADD CONSTRAINT pk_ways PRIMARY KEY (id);
ALTER TABLE ONLY way_nodes ADD CONSTRAINT pk_way_nodes PRIMARY KEY (way_id, sequence_id);
ALTER TABLE ONLY relations ADD CONSTRAINT pk_relations PRIMARY KEY (id);
ALTER TABLE ONLY relation_members ADD CONSTRAINT pk_relation_members PRIMARY KEY (relation_id, sequence_id);
CREATE INDEX idx_node_tags_node_id ON node_tags USING btree (node_id);
CREATE INDEX idx_nodes_geom ON nodes USING gist (geom);
CREATE INDEX idx_way_tags_way_id ON way_tags USING btree (way_id);
CREATE INDEX idx_way_nodes_node_id ON way_nodes USING btree (node_id);
CREATE INDEX idx_relation_tags_relation_id ON relation_tags USING btree (relation_id);
-- Comment these out if the COPY files include bbox or linestring column values.
SELECT AddGeometryColumn('ways', 'bbox', 4326, 'GEOMETRY', 2);
SELECT AddGeometryColumn('ways', 'linestring', 4326, 'GEOMETRY', 2);
-- Comment these out if the COPY files include bbox or linestring column values.
-- Update the bbox column of the way table.
UPDATE ways SET bbox = (
SELECT ST_Envelope(ST_Collect(geom))
FROM nodes JOIN way_nodes ON way_nodes.node_id = nodes.id
WHERE way_nodes.way_id = ways.id
);
-- Update the linestring column of the way table.
UPDATE ways w SET linestring = (
SELECT ST_MakeLine(c.geom) AS way_line FROM (
SELECT n.geom AS geom
FROM nodes n INNER JOIN way_nodes wn ON n.id = wn.node_id
WHERE (wn.way_id = w.id) ORDER BY wn.sequence_id
) c
)
-- Index the way bounding box column.
CREATE INDEX idx_ways_bbox ON ways USING gist (bbox);
CREATE INDEX idx_ways_linestring ON ways USING gist (linestring);
-- Perform database maintenance due to large database changes.
VACUUM ANALYZE;

View File

@ -0,0 +1,141 @@
-- Database creation script for the simple PostgreSQL schema.
-- Drop all tables if they exist.
DROP TABLE IF EXISTS actions;
DROP TABLE IF EXISTS users;
DROP TABLE IF EXISTS nodes;
DROP TABLE IF EXISTS node_tags;
DROP TABLE IF EXISTS ways;
DROP TABLE IF EXISTS way_nodes;
DROP TABLE IF EXISTS way_tags;
DROP TABLE IF EXISTS relations;
DROP TABLE IF EXISTS relation_members;
DROP TABLE IF EXISTS relation_tags;
DROP TABLE IF EXISTS schema_info;
-- Drop all stored procedures if they exist.
DROP FUNCTION IF EXISTS osmosisUpdate();
-- Create a table which will contain a single row defining the current schema version.
CREATE TABLE schema_info (
version integer NOT NULL
);
-- Create a table for users.
CREATE TABLE users (
id int NOT NULL,
name text NOT NULL
);
-- Create a table for nodes.
CREATE TABLE nodes (
id bigint NOT NULL,
version int NOT NULL,
user_id int NOT NULL,
tstamp timestamp without time zone NOT NULL,
changeset_id bigint NOT NULL
);
-- Add a postgis point column holding the location of the node.
SELECT AddGeometryColumn('nodes', 'geom', 4326, 'POINT', 2);
-- Create a table for node tags.
CREATE TABLE node_tags (
node_id bigint NOT NULL,
k text NOT NULL,
v text NOT NULL
);
-- Create a table for ways.
CREATE TABLE ways (
id bigint NOT NULL,
version int NOT NULL,
user_id int NOT NULL,
tstamp timestamp without time zone NOT NULL,
changeset_id bigint NOT NULL
);
-- Create a table for representing way to node relationships.
CREATE TABLE way_nodes (
way_id bigint NOT NULL,
node_id bigint NOT NULL,
sequence_id int NOT NULL
);
-- Create a table for way tags.
CREATE TABLE way_tags (
way_id bigint NOT NULL,
k text NOT NULL,
v text
);
-- Create a table for relations.
CREATE TABLE relations (
id bigint NOT NULL,
version int