What is a Token in C Programming?
Before the compiler can execute a C program, it first reads and analyzes the source code. During this process, the program is divided into small meaningful parts that the compiler can understand. These smallest meaningful parts are called tokens. In simple words, a token is the basic building block of every C program.
Whether you write a simple program or a large software application, every line of C code is made up of different types of tokens such as keywords, identifiers, constants, string literals, operators, and special symbols. The compiler recognizes these tokens before checking the syntax and generating machine code.
How Does the Compiler Identify Tokens?
The process of identifying tokens takes place during the Lexical Analysis phase, which is the first stage of the compilation process. In this phase, the lexical analyzer (scanner) reads the source code from left to right and groups individual characters into meaningful units called tokens.
While scanning the program, the lexical analyzer ignores unnecessary elements such as spaces, tabs, blank lines, and comments because they do not affect the execution of the program. Only meaningful elements are converted into tokens.
Example of Tokens in C
Consider the following C statement:
int age = 20;The compiler does not treat the entire statement as one unit. Instead, it breaks the statement into separate tokens, as shown below.
| Program Element | Token Type |
|---|---|
int | Keyword |
age | Identifier |
= | Operator |
20 | Integer Constant |
; | Special Symbol (Separator) |
Types of Tokens in C Programming
Every valid C program is formed using different types of tokens. Based on their purpose, tokens in the C programming language are divided into the following six categories.
1. Keywords
Keywords are reserved words provided by the C language. They have predefined meanings and cannot be used as variable names, function names, or identifiers.
Examples: int, float, if, return, while
2. Identifiers
Identifiers are user-defined names used to represent variables, functions, arrays, structures, unions, and other program elements.
Examples: age, marks, total, main
3. Constants
Constants are fixed values that remain unchanged throughout the execution of a program.
Examples: 10, 3.14, 'A'
4. String Literals
A string literal is a sequence of characters enclosed within double quotation marks.
Examples: "Hello", "Suraku Academy"
5. Operators
Operators are special symbols that perform operations such as arithmetic, comparison, assignment, logical operations, increment, decrement, and more.
Examples: +, -, *, =, ==, &&
6. Special Symbols (Separators)
Special symbols define the structure of a C program and separate different parts of the code.
Examples: ;, ,, (), {}, []
Quick Revision
- A token is the smallest meaningful unit of a C program.
- Every C program is made up of one or more tokens.
- Tokens are identified during the Lexical Analysis phase.
- Spaces, tabs, blank lines, and comments are ignored during tokenization.
- The six main types of tokens are Keywords, Identifiers, Constants, String Literals, Operators, and Special Symbols.
Summary of All Token Types
The following table provides a quick overview of all token types used in C programming. It is especially useful for revision before competitive examinations and interviews.
| Token Type | Purpose | Example |
|---|---|---|
| Keyword | Reserved words with predefined meanings. | int, char, return |
| Identifier | User-defined names for variables, functions, arrays, etc. | sum, marks, main |
| Constant | Fixed numeric or character values. | 100, 3.14, 'A' |
| String Literal | Sequence of characters enclosed within double quotes. | "Hello" |
| Operator | Performs arithmetic, logical, relational, assignment and other operations. | +, =, ==, && |
| Special Symbol | Separates and organizes program elements. | (), {}, [], ;, , |
Rules for Counting Tokens
Many students make mistakes while counting tokens because they confuse characters with tokens. Remember that a token represents a meaningful unit, not individual letters.
- Each keyword is counted as one token.
- Each identifier is counted as one token.
- Each constant is counted as one token.
- Each operator is counted as one token.
- Each special symbol is counted as one token.
- A complete string enclosed within double quotation marks is considered one string literal token.
- Spaces, tabs, blank lines and comments are not counted as tokens.
Example 1: Count the Number of Tokens
int a = 10;The compiler divides this statement into the following tokens.
| Token | Category |
|---|---|
int | Keyword |
a | Identifier |
= | Operator |
10 | Constant |
; | Special Symbol |
Example 2: Count the Tokens
printf("Hello World");Although the string contains multiple characters, the compiler considers the complete string as a single string literal token.
| Token | Category |
|---|---|
printf | Identifier (Function Name) |
( | Special Symbol |
"Hello World" | String Literal |
) | Special Symbol |
; | Special Symbol |
Common Mistakes While Counting Tokens
- Counting each letter of an identifier separately.
- Counting every character inside a string literal individually.
- Ignoring operators while counting tokens.
- Treating spaces or blank lines as tokens.
- Forgetting that semicolon (
;) is also a token.
Exam Strategy
When solving token-counting questions, first separate the statement into meaningful parts. Then classify each part as a keyword, identifier, constant, operator, string literal or special symbol. This approach helps you avoid mistakes and solve questions quickly in examinations.
Solved Examples: Counting Tokens in C Programs
The best way to understand tokens is by practicing different C statements. The compiler always separates the source code into meaningful units before checking the syntax. Let's solve some commonly asked examples.
Example 1
float salary = 25000.50;| Token | Category |
|---|---|
float | Keyword |
salary | Identifier |
= | Operator |
25000.50 | Floating Constant |
; | Special Symbol |
Example 2
char grade = 'A';| Token | Category |
|---|---|
char | Keyword |
grade | Identifier |
= | Operator |
'A' | Character Constant |
; | Special Symbol |
Example 3
a = b + c * d;| Token | Category |
|---|---|
a | Identifier |
= | Operator |
b | Identifier |
+ | Operator |
c | Identifier |
* | Operator |
d | Identifier |
; | Special Symbol |
Tricky Examples Frequently Asked in Competitive Exams
Some questions appear simple but are designed to confuse students. Understanding these examples will help you avoid common mistakes in examinations.
Example 4
i++;| Token | Category |
|---|---|
i | Identifier |
++ | Increment Operator |
; | Special Symbol |
Notice that ++ is treated as one operator, not two separate + symbols.
Example 5
if(a >= b)| Token | Category |
|---|---|
if | Keyword |
( | Special Symbol |
a | Identifier |
>= | Relational Operator |
b | Identifier |
) | Special Symbol |
>= is counted as one token, not two separate tokens (> and =).Points to Remember for Competitive Exams
- Keywords always have predefined meanings and cannot be used as identifiers.
- Identifiers are created by programmers to represent variables, functions, arrays, structures, and other program elements.
- A complete string enclosed within double quotation marks is counted as one string literal token.
- Operators such as
++,--,==,!=,&&,||,>=, and<=are single tokens. - Comments and white spaces are ignored during lexical analysis.
- Semicolon (
;), comma (,), parentheses, braces, and brackets are also counted as tokens.
Quick Revision
Whenever you are asked to count tokens in a C program, divide the statement into meaningful parts instead of individual characters. This simple technique helps you solve token-counting questions quickly and accurately in exams like UGC NET, SET, PGT, TGT, DSSSB, Computer Instructor, and other Computer Science competitive examinations.
Important Exam Points
- Token is the smallest meaningful unit of a C program.
- Lexical Analysis is the first phase of the compiler.
- Comments and white spaces are ignored during tokenization.
- Every keyword is a token, but every token is not necessarily a keyword.
- Operators like
++,--,==,>=, and&&are each counted as one token. - String literals enclosed within double quotation marks are counted as one token.
Conclusion:
Tokens are the basic building blocks of every C program. Before checking the syntax or generating machine code, the compiler first divides the source code into meaningful tokens during lexical analysis. Understanding different types of tokens is essential because they form the foundation of C programming and compiler design.
For competitive examinations such as UGC NET, SET, PGT, TGT, Computer Instructor, DSSSB, university examinations, and technical interviews, questions related to tokens are frequently asked. Therefore, students should understand not only the definition of tokens but also how to identify and count them correctly in different C statements.
Final Revision
- Token = Smallest meaningful unit of a C program.
- Identified during the Lexical Analysis phase.
- Six main types of tokens are Keywords, Identifiers, Constants, String Literals, Operators, and Special Symbols.
- Comments and white spaces are ignored.
- Practice token-counting questions regularly to improve accuracy in competitive exams.
The more C programs you read and write, the easier it becomes to recognize different types of tokens. Regular practice with examples and previous year questions will strengthen your understanding and help you solve exam questions with confidence.
