Translate Toolkit 2.0.0b5
*************************

*Released on 30 August 2016*

This release contains many improvements and bug fixes. While it contains many
general improvements, it also specifically contains needed changes and
optimizations for the upcoming `Pootle <http://pootle.translatehouse.org/>`_
2.8.0 and `Virtaal <http://virtaal.translatehouse.org>`_ releases.


2.0.0b5 vs 2.0.0b4
==================

- Added new Mozilla's l20n format, including converters
  :doc:`l20n2po </commands/l20n2po>` and :ref:`po2l20n <po2l20n>`
- Added ical format support for Python 3
- Mozilla's lang format always outputs last unit followed by trailing newlines
- :ref:`po2prop <po2prop>` skips first entry in PO file (:issue:`3463`)
- Added Danish valid accelerators characters (:issue:`3487`)
- Updated and pinned requirements
- Removed misleading extra requirements files


Major changes
=============

- Python 3 compatibility thanks to Claude Paroz
- Dropped support for Python 2.6
- Support for new l20n format
- Translate Toolkit can now easily be installed on Windows
- Changes in storage API to expose a more standardized API


Detailed changes
================

Python 3 support
----------------

- Translate Toolkit went through a massive code cleanup looking forward Python
  3 compatibility. There might be quirks that need to be fixed, so please test
  and `report any issue <https://github.com/translate/translate/issues/new>`_
  you might find.
- Python 3.3-3.5 is now supported.


Requirements
------------

- ``lxml`` requirement was raised to 3.5.0 in order to simplify code.
- Updated and pinned requirements
- Removed misleading extra requirements files


Formats and Converters
----------------------

- PO

  - Duplicate entries are now removed by default if no way to handle
    duplicates is specified. Note that converters still default to `msgctxt`.
  - msgid comments (KDE style) are only parsed from msgid now.
  - Fixed parsing of PO files with first entry in unicode
  - Fixed parsing of locations with percent char

- DTD

  - Newlines are now skipped when parsing (:issue:`3390`).
  - Invalid ampersands are not scrubbed anymore.
  - label+accesskey is now only extracted if it is not followed by space.

- Properties

  - Keys can contain delimiters if they are properly wrapped (:issue:`3275`).
  - Fix control characters escaping for utf-8 encoding.
  - :ref:`po2prop <po2prop>` removes fully untranslated units if
    required
  - :ref:`po2prop <po2prop>` skips first entry in PO file (:issue:`3463`)

- Mozilla .lang

  - ``{ok}`` marker is now more cleanly removed
  - Always output last unit followed by trailing newlines

- Mozilla's l20n

  - Added this new format storage class
  - Added new converters :doc:`l20n2po </commands/l20n2po>` and
    :ref:`po2l20n <po2l20n>`

- Android

  - Unknown locales no longer produce failures.
  - Simplify newlines handling as the format now handles \n and newline equally
    (:issue:`3262`)
  - Moved all namespaces to <resources> element.
  - Simplified newlines handling

- ODF

  - :doc:`odf2xliff </commands/odf2xliff>` now extracts all the text
    (:issue:`3239`).

- ts

  - XML declaration is written with double quotes.
  - Self-closing for 'location' elements are not output anymore.

- JSON

  - Output now includes a trailing newline.
  - Unit ordering is maintained (:issue:`3394`).

- ical

  - Enabled this format for Python 3 too.

- TermBase eXchange (TBX)

  - :doc:`tbx2po </commands/tbx2po>` converter added
  - Added basic support for Parts of Speech and term definitions.

- Fixed error when writing back to the same file (:issue:`3419`).


Filters and Checks
------------------

- Added the ability to skip some checks for some languages in specific checkers

  - ``accelerators`` check is now skipped by several Indic languages in
    ``MozillaChecker`` checker.

- LibreOffice checker no longer checks for Python brace format (:issue:`3303`).
- LibreOffice validxml check correctly matches self-closing tags.
- Numbers check now handles non latin numbers. Support for non latin numbers
  has been added for Arabic, Assamese, Bengali and Persian languages.
- Fixed issue that prevented standard checks from being used in Pootle with
  default settings.
- Fixed missing attribute warning displayed when using ``GnomeChecker``,
  ``LibreOfficeChecker`` and ``MozillaChecker`` checkers.


Tools
-----

- :doc:`posegment </commands/posegment>` now correctly segments Japanese
  strings with half width punctuation sign (:issue:`3280`).
- :doc:`pocount </commands/pocount>` now outputs csv header in one line
- ``buildxpi`` was adjusted to current Mozilla needs


Languages
---------

- Fixed plural form for Slovenian and Turkish.
- Added plural forms for Bengali (Bangladesh), Konkani, Kashmiri, Sanskrit and
  Yue (Cantonese).
- Renamed Oriya to Odia.
- Altered Manipuri name to include its most common name Meithei.
- Added language settings for Brazilian Portuguese.
- Added Danish valid accelerators characters (:issue:`3487`).


Setup
-----

- Fixed Inno Setup builds allowing to easily install Translate Toolkit on
  Windows using the ``pip`` installer. Commands are compiled to .exe files.
- Updated installation instructions for Windows


API changes
-----------

- Dropped ``translate.misc.dictutils.ordereddict`` in favor of
  ``collections.OrderedDict``.
- Added encoding handling in base ``TranslationStore`` class exposing a single
  API.
- Encoding detection in ``TranslationStore`` has been improved.
- Standardized UnitClass definition across ``TranslationStore`` subclasses.
- ``translate.misc.multistring.multistring``:

  - Fixed list coercion to text
  - Fixed comparison regression with multistrings (:issue:`3404`).
  - Re-added ``str`` method (:issue:`3428`).
  - Fixed ``__hash__`` (:issue:`3434`).


API deprecation
---------------

- Passing non-ASCII bytes to the ``multistring`` class has been deprecated, as
  well as the ``encoding`` argument to it.
  Applications should always construct ``multistring`` objects by passing
  characters (``unicode`` in Python 2, ``str`` in Python 3), not bytes. Support
  for passing non-ASCII bytes will be removed in the next version.
- ``TxtFile.getoutput()`` and ``dtdfile.getoutput()`` have been deprecated.
  Either call ``bytes(<file_instance>)`` or use the
  ``file_instance.serialize()`` API if you need to get the serialized store
  content of a ``TxtFile`` or ``dtdfile`` instance.


General
-------

- Dropped support for Python 2.6 since it is no longer supported by the Python
  Foundation. Sticking to it was making us difficult to maintain code while we
  move to Python 3.
- Misc docs cleanups.
- Added more tests.
- Increased Python code health.
- Legacy, deprecated and unused code cleansing:

  - Dropped code for no longer supported Python versions.
  - Removed unused code from various places across codebase.
  - The legacy ``translate.search.indexing.PyLuceneIndexer1`` was removed.
  - The deprecated ``translate.storage.properties.find_delimiter()`` was
    removed and replaced by the
    ``translate.storage.properties.Dialect.find_delimiter()`` class method.
  - Python scripts are now available via `console_scripts` entry point, thus
    allowing to drop dummy files for exposing the scripts.


...and loads of general code cleanups and of course many many bugfixes.


Contributors
============

This release was made possible by the following people:

Claude Paroz, Dwayne Bailey, Leandro Regueiro, Michal Čihař, Taras Semenenko,
Ryan Northey, Kai Pastor, Friedel Wolff, Julen Ruiz Aizpuru, Hiroshi Miura,
Thorbjørn Lindeijer, Melvi Ts, Jerome Leclanche, Jakub Wilk, Zibi Braniecki,
Zdenek Juran, Yann Diorcet, Nick Shaforostoff, Christian Lohmaier, beernarrd.

And to all our bug finders and testers, a Very BIG Thank You.
