← Back to catalogue
Research draft

character encoding

vr.tr.character-encoding · INF.KNW

Let an agent explain character encodings and how they work, relay the major standards and legacy code pages from official sources, describe Unicode and UTF-8, and help diagnose encoding problems in general terms.

Thing Registry Information and virtual systems

Research draft, second pass

A second pass drafted this model: the structure a model of this thing needs, and what is known about it in the world. The line under this one says how the second half was obtained - researched against sources, or recalled without web access, in which case nothing here was read anywhere and every claim is a lead to verify. Unreviewed either way.

written by Claude from model knowledge without web access - no source was read, every claim is a lead to verify

Researched by: Claude

Purpose and description

Let an agent explain character encodings and how they work, relay the major standards and legacy code pages from official sources, describe Unicode and UTF-8, and help diagnose encoding problems in general terms.

A system that assigns numeric codes to characters so text can be stored and transmitted digitally, comprising a coded character set that maps characters to code points and an encoding form that represents those code points as bytes, from ASCII and the many national and vendor code pages such as DOS OEM code pages, Mac Icelandic, MacGreek and APL code pages, through ISO 8859 to Unicode with UTF-8, UTF-16 and UTF-32; mismatched encodings produce garbled text known as mojibake.

What it is for: Representing text in computers.

It can be explain the concept; relay standards and code pages; describe Unicode; help diagnose encoding problems.

Distinguishing features

Character to code mapping

Legacy fragmentation

Unicode unification

Byte-level encoding forms

What it looks like

Not a visible object; tables mapping characters to numbers.

Physical character

ASCII characters: 128 count

Unicode code points: 1114112 count - maximum

UTF-8 share of web pages: about 98 percent - W3Techs

How it is recognised

Mapping of characters to codes

ASCII, code pages, ISO 8859, Unicode, UTF-8, UTF-16

Fonts render characters; keyboard layouts input them; ciphers encode for secrecy

Related models

is a kind of - in registry terms

technical standard

is a kind of - in registry terms

code

is a kind of - in registry terms

data type

is unified by - the universal character set

Unicode

In practice

Families and kinds

ASCII and its extensions

IBM and DOS OEM code pages

Windows code pages

Macintosh encodings such as Mac Icelandic and MacGreek

APL code pages

ISO 8859 series

East Asian encodings such as Shift JIS, GB 2312 and Big5

Unicode encoding forms UTF-8, UTF-16 and UTF-32

Identifiers

ISO/IEC 10646 Universal Coded Character Set

RFC 3629 UTF-8

Standards and regulation

Unicode Standard and ISO/IEC 10646

IANA charset registry

ISO 8859 and national standards

Web standards requiring UTF-8

Failure modes and hazards

Mojibake from mismatched encodings

Data loss in lossy conversions

Security issues from encoding confusion

Legacy code page ambiguity

Also called

coded character setMac Icelandic encodingMacGreek encodingcode pageDOS OEM code pageAPL code pageEBCDIC code pagePolish code pagesWindows code pageXJPDBCSUnicode encodingprecomposed hangul encodingdecomposed hangul encodingKOI character encodingsArmenian Standard Code for Information InterchangeTelegraph codecharacter encoding in HTMLSBCSExtended ASCIIBCD character encodingBinary Ordered Compression for UnicodeRISC OS character setsix-bit character codeBrahmic scripts in Unicodevariable-width character encodingdingbat typefaceGOST 10859character codestandard character setJIS encodingShannon–Fano–Elias codingExtended Unix Code

Where this came from

wikidata · CC0 1.0

Source repository Registry entry point

Drafted structure

Bundle to layer to finding to question, as the second pass will find it: 4 bundles · 8 layers · 8 findings · 16 questions.

Understand What character encoding is.

Definition.

Definition

Definition.

Definition

Definition.

  1. What is a character encoding, and how do coded character sets and encoding forms differ? definition
  2. Is the question about encoding, fonts, keyboards or ciphers? boundary

History

From ASCII to Unicode.

History

History.

  1. How did encodings develop from ASCII through code pages such as DOS OEM, Mac and APL code pages to Unicode? definition
  2. Which entry fits the specific encoding? action
Unicode Unicode.

Standards.

Model

Unicode model.

Model

Model.

  1. How do code points, planes, normalisation and encoding forms work in Unicode? provenance
  2. Which references are standard? provenance

UTF-8

UTF-8 and friends.

UTF-8

UTF-8.

  1. How do UTF-8, UTF-16 and UTF-32 encode code points, and why is UTF-8 dominant? provenance
  2. Which entry fits UTF-8? action
Practice Practice.

Practice.

Diagnose

Diagnosing problems.

Diagnose

Diagnose.

  1. How can garbled text be diagnosed and fixed by identifying the encoding? action
  2. Which sources are cited? provenance

Programming

In programming.

Programming

Programming.

  1. How do languages, databases and protocols declare and convert encodings? action
  2. Which entry fits the specific technology? action
Context Legacy and future.

Context.

Legacy

Legacy encodings.

Legacy

Legacy.

  1. Where do legacy code pages still matter, and how are they mapped to Unicode? provenance
  2. Which entry fits code page? action

Future

Evolution.

Future

Future.

  1. How does Unicode continue to evolve with new scripts and emoji? provenance
  2. Which entry fits Unicode Consortium? action

What the second pass must settle

  • Should Unicode and code page be separate primary entries?
  • How should standards be linked?
  • The registry entry has merged aliases naming specific code pages; should they be split off?
Draft as JSON