1. Operator overloading

Operator overloading is a syntactic sugar available in various programming languages (e.g. C++, Python, Kotlin). This feature contributes to cleaner and more expressive code when performing arithmetic-like operations on objects.

For example, when using a Complex class in PHP, you may want to:

1
2
3
4

$a = new Complex(1.1, 2.2);
$b = new Complex(1.2, 2.3);
$c = $a * $b / ($a + $b);

Instead of:

1

$c = $a->mul($b)->div($a->add($b));

Although there is an RFC aiming at providing this feature for PHP, it’s not yet taken into consideration, which means we can’t simply overload operators in userland PHP.

Fortunately, operator overloading can be achieved in native PHP with a little bit of invocation to several Zend APIs, without having to temper with any internal code of PHP itself. The PECL operator extension already does that for us (the releases are out of date, see the git master branch for PHP7 support).

In this article, we will talk about the details of implementing operator overloading in a native PHP extension. We assume that you know the basics of the C/C++ programming language and basics of the Zend implementation of PHP.

2. Opcodes in PHP

Before a PHP script can be executed in Zend VM, it is compiled into arrays of zend_ops. Similar to machine codes, a zend_op consists of an instruction, at most two operands, and the operation result.

1
2
3
4
5
6
7
8
9
10
11
12

struct _zend_op {
    const void *handler;      // Function pointer to opcode handler.
    znode_op op1;             // First operand.
    znode_op op2;             // Second operand.
    znode_op result;          // Instruction result.
    uint32_t extended_value;  // Extra data corresponding to this opline.
    uint32_t lineno;          // Line numper of this opline.
    zend_uchar opcode;        // Instruction code.
    zend_uchar op1_type;      // Type of first operand.
    zend_uchar op2_type;      // Type of second operand.
    zend_uchar result_type;   // Type of instruction result.
};

2.1 Operands

The znode_op is a union storing offset or pointer to the referring target.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

typedef union _znode_op {
    uint32_t      constant;
    uint32_t      var;
    uint32_t      num;
    uint32_t      opline_num;
#if ZEND_USE_ABS_JMP_ADDR
    zend_op      *jmp_addr;
#else
    uint32_t      jmp_offset;
#endif
#if ZEND_USE_ABS_CONST_ADDR
    zval         *zv;
#endif
} znode_op;

As said in zend_compile.h:

On 64-bit systems, less optimal but more compact VM code leads to better performance. So on 32-bit systems we use absolute addresses for jump targets and constants, but on 64-bit systems relative 32-bit offsets.

The ZEND_USE_ABS_JMP_ADDR and ZEND_USE_ABS_CONST_ADDR macros are defined to 0 when PHP is compiled on 64-bit systems, thus znode_op is always 32 bits in size.

2.2 Instructions

The instruction codes are defined in zend_vm_opcodes.h. Operators are converted to corresponding instruction codes when PHP scripts are compiled.

For example, the following assembly-like code represents $c = $a + $b (You can try that yourself by using phpdbg):

1
2

ADD    $a, $b, ~0  # "+" operator 
ASSIGN $c, ~0      # "=" operator

However, not all operators have a corresponding instruction (e.g. negation, greater than, not less than operators). PHP code $c = $a > -$b will be compiled to something like:

1
2
3

MUL        $b, -1, ~0  # Converted to "$b * (-1)" (since PHP 7.3).
IS_SMALLER ~0, $a, ~1  # Converted to "<".
ASSIGN     $c, ~1

2.3 Type of operands

Operand type can be of following:

1
2
3
4
5

#define IS_UNUSED   0
#define IS_CONST    (1<<0)
#define IS_TMP_VAR  (1<<1)
#define IS_VAR      (1<<2)
#define IS_CV       (1<<3) // Compiled variable

• If the operand is not used by the instruction, or the instruction doesn’t generate a result, the type of corresponding znode_op is IS_UNUSED.
• If the operand is a literal, its type is IS_CONST.
• If the operand is the temporary value returned by an expression, its type is IS_TMP_VAR.
• If the operand is a variable known at compile time, its type is IS_CV.
• If the operand is a variable returned by an expression, its type is IS_VAR.

The following PHP code:

1
2
3
4
5

$a = 1;
$a + 1;
$b = $a + 1;
$a += 1;
$c = $b = $a += 1;

Compiles to:

1
2
3
4
5
6
7
8
9
10

#                        (op1     op2     result) type
ASSIGN     $a, 1       #  CV      CONST   UNUSED
ADD        $a, 1,  ~1  #  CV      CONST   TMP_VAR
FREE       ~1          #  TMP_VAR UNUSED  UNUSED
ADD        $a, 1,  ~2  #  CV      CONST   TMP_VAR
ASSIGN     $b, ~2      #  CV      TMP_VAR UNUSED
ASSIGN_ADD $a, 1       #  CV      CONST   UNUSED
ASSIGN_ADD $a, 1,  @5  #  CV      CONST   VAR
ASSIGN     $b, @5, @6  #  CV      VAR     VAR
ASSIGN     $c, @6      #  CV      VAR     UNUSED

We can see that, for an assignment instruction, whether it has a result depends on whether the result is used or not. But for non-assignment instructions, the result is always stored in a temporary variable, even when the result is unused, in case it needs to be freed.

3. Opcode handlers

An opcode handler is a function which executes a zend_op, like CPU which executes a machine instruction. A Zend API provides us with the ability to replace built-in opcode handlers with user-defined ones:

1
2
3

ZEND_API int zend_set_user_opcode_handler(
    zend_uchar            opcode,
    user_opcode_handler_t handler);

Where opcode is the instruction code to be overridden, and handler is the pointer to the user-defined handler function.

1

typedef int (*user_opcode_handler_t) (zend_execute_data *execute_data);

The handler function should accept execute_data pointer as argument, and returns an int indicating the execution status of the handler function, which could be one of the following values:

1
2
3
4
5

#define ZEND_USER_OPCODE_CONTINUE   0
#define ZEND_USER_OPCODE_RETURN     1
#define ZEND_USER_OPCODE_DISPATCH   2
#define ZEND_USER_OPCODE_ENTER      3
#define ZEND_USER_OPCODE_LEAVE      4

In most cases, we only need the two return values explained below:

• ZEND_USER_OPCODE_CONTINUE indicates that the zend_op is successfully executed, and the program should proceed to the next line of opcode.
• ZEND_USER_OPCODE_DISPATCH indicates that we want to fall back to the built-in opcode handler.

Once the handler is set, it will be invoked by the Zend Engine whenever a znode_op with a corresponding instruction is about to be executed. Note that multiple calls to zend_set_user_opcode_handler() will replace old handlers with new ones.

To disable a user-defined opcode handler, pass nullptr to the handler argument.

3.1 Implementing an opcode handler

First, we define a general-purpose handler function template in C++. The handler argument contains the exact business logic of the opcode handler. It accepts three zval pointers (i.e. two operands and result), and return a bool indicating whether the instruction is executed within this handler.

1
2
3
4
5
6
7
8
9
10

template <typename F>
int op_handler(zend_execute_data *execute_data, F handler)
{
    // ... initialization here
    if (!handler(op1, op2, result)) {
        return ZEND_USER_OPCODE_DISPATCH;
    }
    // ... clean up here
    return ZEND_USER_OPCODE_CONTINUE;
}

Then, we initialize the handler function. Fetch the pointer to the current line of opcode from execute_data, and pointers to each operand zval from opline.

1
2
3
4
5
6
7

const zend_op *opline = EX(opline);
zend_free_op free_op1, free_op2;
zval *op1 = zend_get_zval_ptr(opline, opline->op1_type, &opline->op1,
                              execute_data, &free_op1, 0);
zval *op2 = zend_get_zval_ptr(opline, opline->op2_type, &opline->op2,
                              execute_data, &free_op2, 0);
zval *result = opline->result_type ? EX_VAR(opline->result.var) : nullptr;

A operand may be a reference to another zval, we would want to first dereference it before use.

1
2
3
4
5
6

if (EXPECTED(op1)) {
    ZVAL_DEREF(op1);
}
if (op2) {
    ZVAL_DEREF(op2);
}

Now handler can be invoked. Before continuing to the next line of opcode, don’t forget to free the operands (if necessary) and increment EX(opline).

1
2
3
4
5
6
7
8

if (free_op2) {
    zval_ptr_dtor_nogc(free_op2);
}
if (free_op1) {
    zval_ptr_dtor_nogc(free_op1);
}
// No need to free `result` here.
EX(opline) = opline + 1;

At last we can register the handler functions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

int add_handler(zend_execute_data *execute_data)
{
    return op_handler(execute_data, [] (auto zv1, auto zv2, auto rv) {
        if (/* Whether we should overload "+" operator */) {
            // ... do something
            return true;
        }
        return false;
    });
}
// Opcode handlers are usually registered on module init.
PHP_MINIT_FUNCTION(my_extension)
{
    zend_set_user_opcode_handler(ZEND_ADD, add_handler);
}

4. Implementation details when overloading operators

Now we know that operator overloading in PHP can be achieved by setting user-defined opcode handlers. However, we should be careful with some details when implementing these functions, otherwise the operators may not work properly as expected.

4.1 Binary operators

A binary operator takes two operands, and always returns a value. Modification of either operand is allowed, provided that the operand type is IS_CV.

Note that there is no ZEND_IS_GREATER or ZEND_IS_GREATER_OR_EQUAL operator as said in section 2.2. Although the PECL operator extension does some hack with extended_value of zend_op to distinguish whether the opcode is compile from “<” or “>”, it requires patching the PHP source code and may break future compatibility. The recommended alternative solution is shown below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

int is_smaller_handler(zend_execute_data *execute_data) {
    return op_handler(execute_data, [] (auto zv1, auto zv2, auto rv) {
        if (Z_TYPE_P(zv1) == IS_OBJECT) {
            if (__zobj_has_method(Z_OBJ_P(zv1), "__is_smaller")) {
                // Call `$zv1->__is_smaller($zv2)`.
                return true;
            }
        } else if (Z_TYPE_P(zv2) == IS_OBJECT) {
            if (__zobj_has_method(Z_OBJ_P(zv2), "__is_greater")) {
                // Call `$zv2->__is_greater($zv1)`.
                return true;
            }
        }
        return false;
    });
}

4.2 Binary assignment operators

Binary assignment operators behaves similar to non-assignment binary operators, with the exception that it should not generate a result when it is not used (opline->result_type == IS_UNUSED). When you overload these operators, make sure that you never touch the result zval in such circumstances.

The execution result of a binary assignment operator is expected to replace the first operand. However, it is not mandatory and is not done automatically by the Zend Engine.

Code example:

1
2
3
4
5
6
7
8
9
10
11
12
13

int assign_add_handler(zend_execute_data *execute_data) {
    return op_handler(execute_data, [] (auto zv1, auto zv2, auto rv) {
        if (Z_TYPE_P(zv1) == IS_OBJECT) {
            // ... handle addition.
            __update_value(Z_OBJ_P(zv1), add_result);
            if (rv != nullptr) {
                ZVAL_COPY(rv, zv1);
            }
            return true;
        }
        return false;
    });
}

4.3 Unary operators

A unary operator takes one operand (opline->op1), and always returns a value. Modification of the operand is allowed, provided that the operand type is IS_CV.

There’s no opcode for negation operator -$a or unary plus operator +$a, as said in section 2.2, because they are compiled to multiplication with -1 and 1. In cases where they are not expected to behave identically, add some logic to the ZEND_MUL handler to workaround this.

Note that compatibility issues exists between PHP 7.3 and versions below 7.3.

A simple example to workaround the negation operator for all major PHP versions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

int mul_handler(zend_execute_data *execute_data) {
    return op_handler(execute_data, [] (auto zv1, auto zv2, auto rv) {
        if (Z_TYPE_P(zv1) == IS_OBJECT) {
#if PHP_VERISON_ID >= 70300
            if (Z_TYPE_P(zv2) == IS_LONG && Z_LVAL_P(zv2) == -1) {
                // Handle `-$zv1`.
                return true;
            }
#endif
            // Handle `$zv1 * $zv2`.
            return true;
        } else if (Z_TYPE_P(zv2) == IS_OBJECT) {
#if PHP_VERISON_ID < 70300
            if (Z_TYPE_P(zv1) == IS_LONG && Z_LVAL_P(zv1) == -1) {
                // Handle `-$zv2`.
                return true;
            }
#endif
            // Handle `$zv1 * $zv2`.
            return true;
        }
        return false;
    });
}

4.4 Unary assignment operators

Unary assignment operators differ from each other. Post-increment/decrement operators behave identical to non-assignment unary operators, while pre-increment/decrement operators behave identical to binary assignment operators with the exception that they accept only one operand.

This behavior is not hard to understand, as in normal circumstances, the operand of a pre-increment/decrement operator is returned as execution result (with type IS_VAR), while a post-increment/decrement operator has to copy itself to a temporary variable and return it as execution result (with type IS_TMP_VAR).

4.5 Where operator overloading won’t work

Try compiling the following script:

1
2

$a = 2 + 3 * (7 + 9);
$b = 'foo' . 'bar';

You will get:

1
2

ASSIGN $a, 50
ASSIGN $b, "foobar"

You can see that the value of $a and $b is calculated at compile time, and there’s neither arithmetic operations nor string concatenation when the script is running. That is done when an expression contains only literals and operators, which will be recognized by the compiler and evaluated with function zend_const_expr_to_zval(), defined in zend_compile.h. Opcode handlers are fetched via get_binary_op(), get_unary_op(), etc., which hard-codes the built-in handler functions. Therefore, if you overload these operators in your extension, your handlers won’t be invoked.

5. Notes

• For full executable example, you may want to see an implementation of class Complex, which is a part of a work-in-progress PHP extension I’m currently working on.
  • complex.hh, which contains handler functions for class Complex.
  • complex.cc, which implements the class Complex.
  • operators.cc, which implements operator overloading.
  • 002-complex-operators.phpt, a test case.
• Overriding opcode handlers can be used for various purposes other than operator overloading. For example, when you are implementing a profiler, you may want to handle ZEND_INIT_FCALL and ZEND_RETURN.
• Every coin has two sides. Overriding opcode handlers will reduce overall performance of your PHP script, as additional handler functions are called every time when executing a hooked opcode.

1. Properties and hashtables

In statically typed languages, such as C++, properties of objects can be accessed via offset, which is determined at compile time. However, PHP is dynamically typed. There’s no way we can determine which type a zval is until we execute to the exact line of opcode, so it’s impractical trying to access properties via offset.

In the Zend implementation of PHP, properties of an object is stored in a zend_array, aka a hashtable. See the definition of zend_object:

1
2
3
4
5
6
7
8

struct _zend_object {
    zend_refcounted_h           gc;
    uint32_t                    handle;
    zend_class_entry           *ce;
    const zend_object_handlers *handlers;
    HashTable                  *properties;
    zval                        properties_table[1];
};

Buckets of zobj->properties are key-value pairs of property names and values. Whenever a property of a zend_object is being accessed, the property name is applied to the hash function.

“So what’s zobj->properties_table anyway?”

Well, it’s the place where values of default properties are stored, and the closest thing we have to fetching properties by offset. Let’s suppose there’s a userland PHP class defined as below.

1
2
3
4
5
6
7
8
9
10

class Pair
{
    public $first;
    public $second;
    function __construct($first = null, $second = null)
    {
        $this->first = $first;
        $this->second = $second;
    }
}

The $first and $second property of class Pair are defined explicitly in the code, thus, they are considered as default properties. Value of $first is stored in zobj->properties_table[0], and $second in zobj->properties_table[1], which is determined at compile time. So you can see that the number of default properties affect the size of zend_object.

However, that doesn’t mean that those properties can be accessed via offset. Consider the following code:

1
2
3
4
5
6
7
8
9
10
11
12

declare(strict_types=1);        // Use strict typing.
function foo(Pair $bar, $baz = null) : Pair
{
    echo $bar->first;           // No, hashtable is used.
    if ($baz instanceof Pair) {
        echo $baz->second;      // Still no.
    }
    return $bar;
}
echo foo(new Pair())->first;    // Noooo..

After a short tour of the PHP source code or some debugging with GDB, it will dawn upon you, that even the offset of a default property can be determined at compile time in theory, the Zend engine doesn’t do the optimization, not even in the incoming PHP 7.3 (You can RFC it if you like, but believe me, that could be quite a nasty job).

“Wait a sec. I didn’t see a hashtable holding the names of those default properties.”

Of course not. That hashtable is defined in the class entry, whose attributes are determined at compile time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

struct _zend_class_entry {
    char type;
    zend_string *name;
    struct _zend_class_entry *parent;
    int refcount;
    uint32_t ce_flags;
    int default_properties_count;
    int default_static_members_count;
    zval *default_properties_table;
    zval *default_static_members_table;
    zval *static_members_table;
    HashTable function_table;
    HashTable properties_info;
    HashTable constants_table;
    // ...

Whenever you’re trying to access a property, it first search the zobj->ce->properties_info, whose buckets are key-value pairs of default property names and their offsets. If the key does not exist, search zobj->properties. See the source code for details. Behaviors on zend_objects are controlled by zobj->handlers. Unless overriden by native code, those std handler functions will always get called.

Having to perform a second hash search makes accessing dynamic property slower. Meanwhile, as zobj->properties is NULL by default, allocating and initializing a zend_array as container for dynamic properties is also expensive.

For performance comparison, try the following benchmark.

1
2
3
4
5
6
7
8
9
10
11
12
13

// 1. write property
$foo = new Pair(0, 0);
$start = microtime(true);
for ($i = 0; $i < 10000000; ++$i) {
    $foo->first = $i;
}
$duration = microtime(true) - $start;
// 2. call constructor
$start = microtime(true);
for ($i = 0; $i < 1000000; ++$i) {
    $foo = new Pair($i, $i);
}
$duration = microtime(true) - $start;

For the first benchmark, it’s about 32% faster when using a Pair with default properties than one without. For the second one, it’s 41%. (Using PHP 7.2.9 ZTS DEBUG on Darwin, and similar results for other builds of PHP)

2. Access default properties by offset

Well, in userland PHP, there’s nothing more we can do. Just bear in mind that dynamic properties work poorly in performance. Always define properties explicitly if you can.

However, in native context (e.g. in a PHP extension), performance can be improved drastically, as there’s no need for hashtables when accessing default properties.

With Zend API zend_declare_property() and its variants, we can declare default properties in PHP_MINIT_FUNCTION, which will be called only once when the extension gets loaded. Then we can access the values of those properties by zobj->properties_tables[offset]. It’s recommended that you just use macro OBJ_PROP_NUM().

See the following example for a Pair::__construct():

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

PHP_METHOD(Pair, __construct)
{
    zval* first;
    zval* second;
    ZEND_PARSE_PARAMETERS_START(2, 2)
        Z_PARAM_ZVAL(first)
        Z_PARAM_ZVAL(second)
    ZEND_PARSE_PARAMETERS_END();
    // Get current `zend_object`.
    zend_object* zobj = Z_OBJ_P(getThis());
    // Get pointer to properties by offset.
    zval* pair_first = OBJ_PROP_NUM(zobj, 0);
    zval* pair_Second = OBJ_PROP_NUM(zobj, 1);
    // Update `$this->first`.
    Z_TRY_ADDREF_P(first);
    zval_ptr_dtor(pair_first);
    ZVAL_COPY_VALUE(pair_first, first);
    // Update `$this->second`.
    Z_TRY_ADDREF_P(second);
    zval_ptr_dtor(pair_second);
    ZVAL_COPY_VALUE(pair_second, second);
}

1. Bug encountered

Some time ago, I switched to ZSH, using oh-my-zsh‘s default theme, robbyrussell. On my very first attempt to use it, I discovered a mysterious bug. For example:

When auto-completing with TAB, the current line will shift back one character. It’s harmless in this occasion, however, if you are auto-completing in the middle of a long command line, this will probably mess thing up, and you have to rewrite the entire command.

After that, I tested this in GNOME terminal, and it worked fine.

2. Bug located

I looked through the GitHub issue page, and found out that no one else seem to have encountered a similar issue. Everyone uses auto-complete, if there’s really a bug, how come it’s never mentioned? Perhaps there’s a bug in the terminal emulator of the SSH client I was using.

The arrow character at the beginning of the line seems larger in Bitvise xterm than that in GNOME terminal, so I highlighted them and made a comparison.

Now we see the difference. In Bitvise xterm the arrow character is displayed two characters wide. In GNOME terminal, it’s the same width as ASCII characters. The difference in display was probably due to the two terminals using different fonts. However, this is not the cause of the bug, either.

When we echo '➜' | xxd we can see that the UTF-8 encoded right arrow is 0xE29E9C. Now we vim ~/.oh-my-zsh/themes/robbyrussell.zsh-theme and replace arrows with a Chinese character “哈”(UTF-8 0xE59388), open a new terminal, and try again.

Auto-complete behavior with the Chinese character is normal, while the arrow character(having the same size and display width) isn’t. Why? Now type an arrow into the terminal, and try to delete it with backspace.

Uh-oh, the cursor only shifted back one character, and the arrow is still there. Now we found the bug. Bitvise xterm treated the arrow character as one character wide, but display it with two characters wide.

In our first case, tab auto-completing will relocate the line using an offset depending on how many characters are before it. The arrow character is treated as one character wide, so the offset is 5 instead of 6. That’s why the command line is shifted back one character.

3. Workaround

Workaround to this bug is simple. Just prevent using such characters in command line. To stop auto-completing from going wrong, change the theme configuration file and replace the arrow with something else. For example, I changed it into %(!.#.$).

Now the problem is solved.

1. Deducing whether return value will be used

When we call a PHP function, it may return a value. However, we may not always use it. In this case, whatever the function does for returning that value is a waste of CPU cycles. In some rare cases, to improve performance, we may want to deduce whether the return value will be used.

Unfortunately, there’s no way to do that in userland PHP, but in a PHP extension, that’s quite easy. There’s a macro defined in zend.h.

1
2
3
4

#define USED_RET() \
    (!EX(prev_execute_data) || \
     !ZEND_USER_CODE(EX(prev_execute_data)->func->common.type) || \
     (EX(prev_execute_data)->opline->result_type != IS_UNUSED))

We can see that the return value is considered used if at least one of the following conditions is met.

• There’s no previous execute data.
• The function which called this function is not defined in userland PHP.
• The result type stored in the opline of previous execute data is not IS_UNUSED.

1.1 No previous execute data?

Well, the global scope has no previous execute data. You can return from global scope so that another script who includes this script will get the return value. For example:

1
2
3

// foo.php
<?php
return 'foo';

1
2
3

// bar.php
<?php
$bar = include 'foo.php';

Invoking USED_RET() in global scope will always get true. However, that doesn’t concern us, because the native functions we implement in a PHP extension is never in global scope.

1.2 Not userland PHP?

The func property of previous execute data stores the pointer to the zend_function from which the current function is being invoked. For example, we define two native functions in an extension like this:

1
2
3
4
5
6
7
8
9
10
11
12

PHP_FUNCTION(foo)
{
    RETVAL_LONG(EX(prev_execute_data)->func->common.type);
}
PHP_FUNCTION(bar)
{
    zval retval, foo;
    ZVAL_STRING(&foo, "foo");
    call_user_function(EX(function_table), NULL, &foo, &retval, 0, NULL);
    zval_ptr_dtor(&foo);
    RETVAL_LONG(Z_LVAL(retval));
}

Then, run the following script, and you will get “2 4 1” as output.

1
2
3
4

$type_a = foo();
$type_b = eval('return foo();');
$type_c = bar();
echo "$type_a $type_b $type_c", PHP_EOL;

Let’s take a look at zend_compile.c:

1
2
3
4
5
6
7
8

#define ZEND_INTERNAL_FUNCTION              1
#define ZEND_USER_FUNCTION                  2
#define ZEND_OVERLOADED_FUNCTION            3
#define	ZEND_EVAL_CODE                      4
#define ZEND_OVERLOADED_FUNCTION_TEMPORARY  5
/* A quick check (type == ZEND_USER_FUNCTION || type == ZEND_EVAL_CODE) */
#define ZEND_USER_CODE(type) ((type & 1) == 0)

So it’s clear that if the previous scope is in a function defined in PHP, the type property yields ZEND_USER_FUNCTION. If in evaled code, ZEND_EVAL_CODE. If in a native function, ZEND_INTERNAL_FUNCTION. The ZEND_USER_CODE() macro is used to detect whether a function type is either ZEND_USER_FUNCTION or ZEND_EVAL_CODE. If so, that function is considered userland PHP.

The USED_RET() macro will always yield true if the function which invoked the current function is not defined in userland PHP. Why? Because there’s no way to check whether a native function(like bar() in our example) will use the return value or not.

1.3 Result type of opline?

Finally, we check whether return value is used by checking the result_type property of the opline from the calling scope. You can understand opline as “line of opcode”, which is the current line of opcode executed from the calling scope.

The result_type can be one of the following values:

1
2
3
4
5

#define IS_CONST    (1<<0)
#define IS_TMP_VAR  (1<<1)
#define IS_VAR      (1<<2)
#define IS_UNUSED   (1<<3)  /* Unused variable */
#define IS_CV       (1<<4)  /* Compiled variable */

If the value of result_type is IS_UNUSED, we are certain that the return value of the called function will not be used.

2. Usage

As said in the previous section, we can use USED_RET() when the return value of our native function is not mandatory, and may cause performance overhead. Some built-in functions of PHP already take advantage of that macro, for example, functions which manipulate the internal pointer of a zend_array, such as next(), prev(), end(), reset(). If the user just want to do something to the pointer without fetching the corresponding value of the array, the function won’t do the fetch & return job.

For example, in array.c:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

/* {{{ proto mixed next(array array_arg)
   Move array argument's internal pointer to the next element and return it */
PHP_FUNCTION(next)
{
    HashTable *array;
    zval *entry;
    
    ZEND_PARSE_PARAMETERS_START(1, 1)
        Z_PARAM_ARRAY_OR_OBJECT_HT_EX(array, 0, 1)
    ZEND_PARSE_PARAMETERS_END();
    
    zend_hash_move_forward(array);
    
    if (USED_RET()) {
        if ((entry = zend_hash_get_current_data(array)) == NULL) {
            RETURN_FALSE;
        }
        
        if (Z_TYPE_P(entry) == IS_INDIRECT) {
            entry = Z_INDIRECT_P(entry);
        }
        ZVAL_DEREF(entry);
        ZVAL_COPY(return_value, entry);
    }
}
/* }}} */

This macro is available in all current versions of PHP 7. Feel free to use it in your extension, if necessary.

3. How to do that in userland PHP?

You can’t do that in userland PHP, not without the help of a native function. Write a function in your extension like this, and it’s ready to use:

1
2
3
4
5
6
7
8
9
10
11

PHP_FUNTION(used_ret)
{
    zend_execute_data* ex = EX(prev_execute_data);
    if (!ex)
        RETURN_TRUE;
    // Fetching the previous execute data of previous execute data.
    ex = ex->prev_execute_data;
    if (!ex || !ZEND_USER_CODE(ex->func->common.type))
        RETURN_TRUE;
    RETVAL_BOOL(ex->opline->result_type != IS_UNUSED);
}

Just that simple :) Now we can test it with a PHP script.

1
2
3
4
5
6
7
8
9
10
11

function foo()
{
    if (used_ret()) {
        echo 'Used.', PHP_EOL;
    } else {
        echo 'Not used.', PHP_EOL;
    }
    return 0;
}
foo();
$bar = foo();

Expected output:

Not used.
Used.

1. Copy-on-Write mechanism in PHP

PHP beginners may read something like this in books when learning the basics of PHP.

1
2
3

$foo = ['foo' => 'baz']; // ['foo' => 'baz']: refcount = 1
$bar = $foo;             // ['foo' => 'baz']: refcount = 2
$bar['foo'] = 'bar';     // ['foo' => 'baz']: refcount = 1, ['foo' => 'bar']: refcount = 1

PHP arrays are refcounted, so when multiple zvals are referring to a same zend_array, it won’t be copied, however, its refcount is incremented. When any of these arrays is going to be modified, Zend Engine checks the zend_array‘s refcount. If it’s not 1, create a duplicate of the array, modify it, assign the zval to it, and decrement the original array. This is called separation.

The same is true of passing an array to a function parameter by value.

1
2
3
4
5
6

function add_foo($arr) { // ['foo' => 'baz'] : refcount = 2
    $arr['foo'] = 'bar'; // ['foo' => 'baz'] : refcount = 1, ['foo' => 'bar'] : refcount = 1
    // ...
}
$arr = ['foo' => 'baz']; // ['foo' => 'baz'] : refcount = 1
add_foo($arr);           // ['foo' => 'baz'] : refcount = 1

However, PHP objects are treated differently. You can modify a zend_object‘s property_table without separation.

1
2
3
4
5

$arr = ['foo' => 'baz'];
$foo = new ArrayObject($arr); // obj(['foo' => 'baz']) : refcount = 1
$bar = $foo;                  // obj(['foo' => 'baz']) : refcount = 2
$foo[] = 'abc';               // obj(['foo' => 'baz', 'abc']) : refcount = 2
add_foo($bar);                // obj(['foo' => 'bar', 'abc']) : refcount = 2

2. Messing around with arrays in PHP extensions

In PHP extensions, separation should be done manually. Careless or unexperienced PHP extension developers may unintentionally mess thing up. Consider the following code.

1
2
3
4
5
6
7
8
9
10
11

PHP_FUNCTION(add_foo_internal)
{
    zend_array* arr;
    ZEND_PARSE_PARAMETERS_START(1, 1)
        Z_PARAM_ARRAY(arr)
    ZEND_PARSE_PARAMETERS_END();
    zval bar;
    ZVAL_STRING(&bar, "bar");
    zend_hash_str_add(arr, "foo", sizeof "foo" - 1, &bar);
    // ...
}

Looks like pretty legitimate code. However, calling this function may bring you catastrophes.

1
2
3

$foo = ['bar' => 'baz']; // ['foo' => 'baz']: refcount = 1
$bar = $foo;             // ['foo' => 'baz']: refcount = 2
add_foo_internal($bar);  // ['foo' => 'bar']: refcount = 2

“Hey, wait a second, add_foo_internal() behaves just like add_foo() which accepts the parameter by reference. That’s not really a bad thing, after all?”

You’re WRONG.

As a result of separation, if add_foo() accepts the parameter by reference, only $bar will become ['foo' => 'bar'], while $foo will remain the same. However, the add_foo_internal() function actually modifies the zend_array by force, thus changing the values of all the zvals referencing this zend_array, which violates the copy-on-write mechanism of PHP.

Most of the times, that is not what you want. Even if it doesn’t break your extension, it will certainly break userland PHP.

3. HT_ASSERT_RC1 macro in PHP 7.2+

Since PHP 7.2, there’s a HT_ASSERT_RC1() macro defined in zend_hash.c, which is invoked before performing any modification on a zend_array in all hashtable APIs defined in this source file.

1
2
3
4
5
6
7
8

#if ZEND_DEBUG
#define HT_ASSERT(ht, expr) \
    ZEND_ASSERT((expr) || ((ht)->u.flags & HASH_FLAG_ALLOW_COW_VIOLATION))
#else
#define HT_ASSERT(ht, expr)
#endif
#define HT_ASSERT_RC1(ht) HT_ASSERT(ht, GC_REFCOUNT(ht) == 1)

This assertion ensures that whenever a zend_array is modified, its refcount must be 1. If you are using a debug build of PHP, calling add_foo_internal() will trigger assertion failure like this:

php: /opt/php-7.2.4/Zend/zend_hash.c:549: _zend_hash_add_or_update_i: Assertion `((ht)->gc.refcount == 1) || ((ht)->u.flags & (1<<6))’ failed.

UPGRADING.INTERNALS didn’t mention this new feature, but it’s one of the things I like most in PHP 7.2. The rather “pedantic” inspection of copy-on-write violation on PHP arrays is indeed helpful for PHP extension developers. It helped me found out some potentially hazardous bugs in some of my old extensions.

“All right, I know PHP arrays are refcounted, and should be separated before modifying. But in some cases, I have to modify a zend_array whose refcount isn’t 1. What should I do?”

Just add a flag in the zend_array you’d like to mess with, by invoking HT_ALLOW_COW_VIOLATION() macro.

1
2
3
4
5

#if ZEND_DEBUG
#define HT_ALLOW_COW_VIOLATION(ht) (ht)->u.flags |= HASH_FLAG_ALLOW_COW_VIOLATION
#else
#define HT_ALLOW_COW_VIOLATION(ht)
#endif

However, even if you know what you’re doing, “having to do” such a thing is usually a code smell and caused by bad design. It is recommended that you refactor the code and avoid doing so.

I opened a column in the Laravel-China community, PHP Extension Cookbook. It’s not much, but if you look forward to learning PHP extension development, this column can be of help.

I’ve just posted the very first article yesterday, mainly explaining how to write config.m4 script for your extension. And there’s going to be many more.

If there’s something wrong or badly explained in the articles, feel free to email me for correction. And if you’re also adept at developing PHP extensions, you’re welcome to contribute to this column.

Have a great time hacking with PHP, y’all :)

After an unsuccessful attempt to compile my extension with the latest PHP, I discovered that you can no longer directly update GC_REFCOUNT(). As in zend_types.h, macros concerning GC refcount are defined as below:

1
2
3
4

#define GC_REFCOUNT(p)          zend_gc_refcount(&(p)->gc)
#define GC_SET_REFCOUNT(p, rc)  zend_gc_set_refcount(&(p)->gc, rc)
#define GC_ADDREF(p)            zend_gc_addref(&(p)->gc)
#define GC_DELREF(p)            zend_gc_delref(&(p)->gc)

Meanwhile, in PHP 7.2 and older versions:

1

#define GC_REFCOUNT(p)          (p)->gc.refcount

Here’s a simple workaround for compatibility.

1
2
3
4
5

#if PHP_VERSION_ID < 70300
#define GC_ADDREF(p)            ++GC_REFCOUNT(p)
#define GC_DELREF(p)            --GC_REFCOUNT(p)
#define GC_SET_REFCOUNT(p, rc)  GC_REFCOUNT(p) = rc
#endif

This change in internal API was intended to eliminate race-conditions in multi-thread applications, as mentioned in this pull request.

Other notable API changes can be found here, with which you can make your extension compatible with PHP 7.3.

Since PHP 7.0, a new Zend API was implemented for faster parameter parsing.

For example, if your function accepts an integer parameter foo, then the code may look like this.

1
2
3

ZEND_PARSE_PARAMETERS_START(1, 1)
    Z_PARAM_LONG(foo)
ZEND_PARSE_PARAMETERS_END();

However, if your extension is written in C++, the compiler will complain and refuse to compile.

You’ll get error message like:

1

error: invalid conversion from 'int' to 'zend_expected_type {aka _zend_expected_type}' [-fpermissive]

Confused? Let’s take a look at the macro definition, which is located in zend_API.h

1
2
3
4
5
6
7
8
9
10
11

#define ZEND_PARSE_PARAMETERS_START_EX(flags, min_num_args, max_num_args) do { \
        const int _flags = (flags); \
        int _min_num_args = (min_num_args); \
        int _max_num_args = (max_num_args); \
        int _num_args = EX_NUM_ARGS(); \
        int _i; \
        zval *_real_arg, *_arg = NULL; \
        zend_expected_type _expected_type = IS_UNDEF; \
        char *_error = NULL; \
        zend_bool _dummy; \
// Some more code...

We can see on line 8 Zend’s trying to initialize an enum zend_expected_type with value 0, which is forbidden in C++. In C++, you should either explicitly cast it with static_cast or initialize using a corresponding enum value.

Fortunately the value 0 is defined in macro IS_UNDEF (why this??), you can just redefine it instead of sed zend_API.h in your config.m4 script.

Now your code may look like this.

1
2
3
4
5
6
7

#undef IS_UNDEF
#define IS_UNDEF Z_EXPECTED_LONG // Which is zero
ZEND_PARSE_PARAMETERS_START(1, 1)
    Z_PARAM_LONG(foo)
ZEND_PARSE_PARAMETERS_END();
#undef IS_UNDEF
#define IS_UNDEF 0

Ugly, but your code will compile. Cheers :)

P.S. The latest PHP 7.2 still have this problem. Perhaps I should report this issue to the PHP internals guys.

During my attempt to execute a shell script provided by the manufacturer, I got a couple o’ annoying error outputs(shown below).

Going through the script several times, it seemed that there were nothing wrong with it. But it suddenly dawned on me that I was using Chinese as system language for my CentOS 7.

Then I found this. It’s obvious that grep Disk may not work as expected.

1
2

SIZE=`fdisk -l $DRIVE | grep Disk | awk '{print $5}'`
CYLINDERS=`echo $SIZE/255/63/512 | bc`

Gotcha..

The script worked after I changed system language to English. Perhaps we should never use Chinese(as well as other languages beside English) as system language. :)

P.S. In fact, you can do something like export LC_ALL="en_US.UTF-8" to switch locale insead of language, if you really wanna use other languages.

1. List of plans

• Learn Java and Spring Boot.
• Maintenance for TJUBBS backend. Including bug fix and adding minor functionalities.
• Finish implementing all functionalities in php-asio and make a stable release.
• Continue learning C# and Unity3D. Finish the VR project.
• Continue learning data mining.

2. Optional

• Write a client of YAP, and test the sanity of the demo server.
• Rewrite RARBG crawler using amphp/aeyrs and amphp/artax for better performance.
• Write an asynchronous MySQL client for Workerman.